Neural network model training method, data processing method and device

Abstract

This application discloses a training method, data processing method, and apparatus for a neural network model in the field of artificial intelligence. The training method comprises: training the neural network model based on training data, wherein the activation function of the neural network model includes at least one piecewise function, the piecewise function including multiple trainable parameters, and updating the multiple trainable parameters of the at least one piecewise function during the training process. The method of the application can obtain an activation function suitable for the neural network model, thereby improving the performance of the neural network model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a training method, data processing method, and device for a neural network model. Background Art

[0002] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0003] A deep neural network is a feedforward neural network with a deep structure and is one of the representative algorithms of deep learning. Deep neural networks possess powerful feature representation capabilities and are widely used in processing and analyzing various media signals, including images, video, and speech. Activation functions are a crucial component of deep neural networks. Activation functions are typically placed after convolutional or fully connected layers, introducing nonlinear operations into deep neural networks. This allows them to better fit nonlinear functions and solve complex real-world problems. Activation functions have a significant impact on various aspects of model performance, including accuracy and convergence speed.

[0004] Most of the existing activation functions are fixed-parameter activation functions, or the space for parameter variation is small. Such activation functions cannot guarantee good results on all models and tasks. In addition, activation functions can also be searched through reinforcement learning algorithms. The Swish function is a relatively general activation function obtained through reinforcement learning algorithm search, which can be applied to a variety of neural network models. However, in the process of searching for activation functions through reinforcement learning algorithms, the search space is limited to predefined unary and binary functions, which is relatively limited; moreover, the sampling-based reinforcement learning algorithm searches for activation functions, and the search efficiency is low, making it difficult to search for the optimal activation function form for each task and model. The Swish function is only used in the model as a relatively general activation function, and it cannot be guaranteed to be the optimal activation function for all models and tasks.

[0005] Therefore, how to obtain an activation function suitable for the neural network model and improve the performance of the neural network model has become an urgent problem to be solved. Summary of the Invention

[0006] The present application provides a training method, a data processing method and a device for a neural network model, which can obtain an activation function suitable for the neural network model and improve the performance of the neural network model.

[0007] In a first aspect, a training method for a neural network model is provided, the method comprising: obtaining training data; training the neural network model based on the training data, wherein the activation function of the neural network model includes at least one piecewise function, and the piecewise function includes multiple trainable parameters; and updating multiple trainable parameters of at least one piecewise function during the training process of the neural network model to obtain a target neural network model.

[0008] According to the solution of the embodiment of the present application, at least one piecewise function is used as the activation function. By adjusting the multiple trainable parameters of the at least one piecewise function, a variety of function forms can be fitted. That is, the search space for the activation function is larger, which is conducive to obtaining a more optimal activation function. At the same time, during the model training process, the multiple trainable parameters of the piecewise function are updated through parameter gradients, which reduces the operating overhead and improves the efficiency of constructing the activation function and training the model. This makes it possible to search for activation functions for different models, obtain activation functions suitable for different models, and improve the performance of the model.

[0009] The type of training data depends on the task of the neural network model. For example, if the neural network model is used for image processing tasks, the training data can be images. Specifically, image processing tasks include image classification, image detection, image segmentation, or image generation. For another example, if the neural network model is used for text processing tasks, the training data can be text. Specifically, text processing tasks include text recognition or text translation. For another example, if the neural network model is used for speech processing tasks, the training data can be speech data. Specifically, speech processing tasks include speech recognition.

[0010] During a training process of training the neural network model, the multiple trainable parameters are updated based on the gradients of the multiple trainable parameters.

[0011] In combination with the first aspect, in certain implementations of the first aspect, at least one piecewise function is a piecewise linear function, and the parameters of at least one piecewise function include one or more of the following: the number of dividing points, the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0012] According to the solution in the embodiment of the present application, the above-mentioned parameter combination only includes two slope values. In the subsequent parameter update process, the change in the function value corresponding to a dividing point only affects the functions of the two intervals adjacent to the dividing point. The impact range is small, which can improve the training efficiency and the performance of the model after training.

[0013] Furthermore, using a piecewise linear function allows for fitting various function forms, providing a larger search space for the activation function, thereby increasing the upper limit of the precision of the activation function found. Furthermore, compared to using a piecewise nonlinear function as an activation function, using a piecewise linear function as an activation function can improve computational efficiency and reduce time overhead.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the training process for training the neural network model includes a first stage and a second stage, the first stage is executed before the second stage, and multiple trainable parameters of at least one piecewise function are updated during the training process for training the neural network model, including: in the second stage, updating the multiple trainable parameters of at least one piecewise function according to the gradients of the multiple trainable parameters in the at least one piecewise function; wherein the initial values ​​of the right boundary and the left boundary in the at least one piecewise function in the second stage are determined according to the distribution of features input to the at least one piecewise function in the first stage.

[0015] According to the solution of the embodiment of the present application, the values ​​of the left and right boundaries of the activation function are adjusted based on the distribution of the input features counted in the first stage. In this way, the left and right boundaries can match the actual distribution of the input features, so that the input features fall between the left and right boundaries as much as possible. In this way, more parameters in the activation function can be involved in the training process, effectively utilizing multiple segments in the piecewise function, or in other words, the parameters of the activation function can be effectively adjusted using training data, thereby stabilizing the training process, improving the accuracy of the activation function, and further improving the performance of the model and training efficiency.

[0016] In combination with the first aspect, in some implementations of the first aspect, in the first stage, multiple trainable parameters in at least one piecewise function remain unchanged.

[0017] According to the solution of the embodiment of the present application, the parameters of the piecewise function are fixed in the first stage to avoid the impact of changes in the parameters of the piecewise function on the distribution of input features, so that a more accurate distribution of input features can be obtained, which is conducive to the subsequent adjustment of the parameters of the piecewise function.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the distribution of features input into at least one piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, the predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula:

[0020]

[0021] Among them, R mean_j represents the predicted average value of the feature obtained after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, R std_j Represents the predicted standard deviation of the feature obtained after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, R mean_0 represents the initial value of the predicted mean of the feature, R std_0 represents the initial value of the predicted standard deviation of the feature, R mean_0 =0, R std_0 =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0022] a is a positive number less than 1, and b is a positive number less than 1. For example, a is 0.9; b is 0.9.

[0023] In combination with the first aspect, in certain implementations of the first aspect, an initial value of a right boundary RB in at least one piecewise function in the second stage satisfies the following formula:

[0024] RB=R mean +c*R std ;

[0025] The initial value of the left boundary LB in at least one piecewise function in the second stage satisfies the following formula:

[0026] LB=R mean -c*R std ;

[0027] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R stdrepresents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the multiple trainable parameters in at least one piecewise function include: a right boundary, a left boundary, a slope on the interval with the largest domain, a slope on the interval with the smallest domain, or a function value corresponding to a dividing point.

[0029] In other words, the number of cutoff points, N, is treated as a hyperparameter. During training, the number of cutoff points is not updated, while the other parameters of the piecewise function are updated. This ensures that the number of parameters of the piecewise function remains constant, reduces the complexity of updating the piecewise function, and ensures the stability of model training.

[0030] In combination with the first aspect, in some implementations of the first aspect, the number of segments of the piecewise function is any value between 6 and 18.

[0031] In a second aspect, a data processing method is provided, which includes: obtaining data to be processed, the data to be processed including image data, voice data or text data; using a target neural network to process the data to be processed to obtain a processing result of the data to be processed; wherein the target neural network model is obtained by training the neural network model based on the training data, the activation function of the neural network model includes at least one piecewise function, the activation function of the target neural network model includes at least one target piecewise function, and the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model.

[0032] The target neural network model can be obtained by using the training method of the neural network model in the first aspect. The target neural network model can be a trained neural network model, and the trained neural network model can be used to process the data to be processed.

[0033] In this application, since the target neural network model is obtained by adopting the method of the first aspect mentioned above, it is more in line with or close to the application requirements of the neural network model. Using such a neural network model for data processing can achieve better processing effects.

[0034] According to the solution of the embodiment of the present application, at least one piecewise function is used as the activation function. By adjusting the multiple trainable parameters of the at least one piecewise function, a variety of function forms can be fitted. That is, the search space for the activation function is larger, which is conducive to obtaining a more optimal activation function. At the same time, during the model training process, the multiple trainable parameters of the piecewise function are updated through parameter gradients, which reduces the operating overhead and improves the efficiency of constructing the activation function and training the model. This makes it possible to search for the activation function for a specific model, obtain a suitable activation function, and improve the performance of the target neural network model.

[0035] In combination with the second aspect, in certain implementations of the second aspect, at least one piecewise function is a piecewise linear function, and the parameters of at least one piecewise function include one or more of the following: the number of dividing points, the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the training process for training the neural network model includes a first stage and a second stage, the first stage is executed before the second stage, and the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model, including: the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function according to the gradients of the multiple trainable parameters in the piecewise function in the second stage, wherein the initial values ​​of the right boundary and the left boundary in the piecewise function in the second stage are determined according to the distribution of features input to the piecewise function in the first stage.

[0037] In combination with the second aspect, in some implementations of the second aspect, in the first stage, multiple trainable parameters in at least one piecewise function remain unchanged.

[0038] In combination with the second aspect, in certain implementations of the second aspect, the distribution of features input into the piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0039] In conjunction with the second aspect, in certain implementations of the second aspect, the predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula:

[0040]

[0041] Among them, R mean_j represents the predicted average value of the feature obtained after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, Rstd_j Represents the predicted standard deviation of the feature obtained after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, R mean_0 represents the initial value of the predicted mean of the feature, R std_0 represents the initial value of the predicted standard deviation of the feature, R mean_0 =0, R std_0 =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0042] In conjunction with the second aspect, in certain implementations of the second aspect, an initial value of a right boundary RB in at least one piecewise function in the second stage satisfies the following formula:

[0043] RB=R mean +c*R std ;

[0044] The initial value of the left boundary LB in at least one piecewise function in the second stage satisfies the following formula:

[0045] LB=R mean -c*R std ;

[0046] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R std represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

[0047] In combination with the second aspect, in certain implementations of the second aspect, the multiple trainable parameters in at least one piecewise function include: the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0048] In combination with the second aspect, in some implementations of the second aspect, the number of segments of the piecewise function is any value between 6 and 18.

[0049] In a third aspect, a training device for a neural network model is provided, the device comprising a module or unit for executing the method in the above-mentioned first aspect and any one of the implementations of the first aspect.

[0050] In a fourth aspect, a data processing device is provided, which includes a module or unit for executing the method in the above-mentioned second aspect and any one of the implementations of the second aspect.

[0051] It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect, third aspect and fourth aspect.

[0052] In a fifth aspect, a training device for a neural network model is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0053] The processor in the fifth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). Among them, the TPU is a dedicated integrated circuit for artificial intelligence accelerators fully customized by Google for machine learning.

[0054] In the sixth aspect, a data processing device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the second aspect and any one of the implementation methods of the second aspect.

[0055] The processor mentioned in the sixth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processor (NPU), a tensor processing unit (TPU), and so on. The Tensor Processing Unit (TPU) is a custom AI accelerator application-specific integrated circuit (ASIC) developed by Google for machine learning.

[0056] In a seventh aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first aspect or the second aspect.

[0057] In an eighth aspect, a computer program product comprising instructions is provided, which, when run on a computer, enables the computer to execute the method in any one of the implementations of the first or second aspects above.

[0058] In a ninth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementations of the first or second aspect.

[0059] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods of the first aspect or the second aspect.

[0060] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided by an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a partial structure of a neural network model;

[0063] Figure 3 A schematic diagram of the structure of a system architecture provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of another application scenario provided by an embodiment of the present application;

[0068] Figure 8 A schematic flow chart of a training method for a neural network model provided in an embodiment of the present application;

[0069] Figure 9 A schematic diagram of a piecewise linear function provided in an embodiment of the present application;

[0070] Figure 10 A schematic flow chart of the forward propagation process of a piecewise linear function provided in an embodiment of the present application;

[0071] Figure 11 A schematic diagram of an initialization form of an activation function provided in an embodiment of the present application;

[0072] Figure 12 A schematic flow chart of a training method for a neural network model provided in an embodiment of the present application;

[0073] Figure 13 A schematic diagram comparing different activation functions provided in the embodiments of the present application;

[0074] Figure 14 is a schematic flow chart of a data processing method provided in an embodiment of the present application;

[0075] Figure 15 is a schematic block diagram of a training device for a neural network model provided in an embodiment of the present application;

[0076] Figure 16 is a schematic block diagram of a data processing device provided in an embodiment of the present application;

[0077] Figure 17 is a schematic block diagram of a training device for a neural network model provided in an embodiment of the present application;

[0078] Figure 18 It is a schematic block diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solution in this application will be described below with reference to the accompanying drawings.

[0080] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0081] The following is a detailed explanation of the above artificial intelligence thematic framework from the two dimensions of "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

[0082] The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom."

[0083] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0084] (1) Infrastructure:

[0085] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and provides support through a basic platform.

[0086] The infrastructure can communicate with the outside world through sensors, and the computing power of the infrastructure can be provided by smart chips.

[0087] The smart chips here can be hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), and field programmable gate array (FPGA).

[0088] The basic platform of the infrastructure can include related platform guarantees and support such as distributed computing framework and network, and can include cloud storage and computing, interconnected networks, etc.

[0089] For example, for infrastructure, data can be obtained through sensors and external communication, and then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0090] (2) Data:

[0091] The data layer above the infrastructure represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0092] (3) Data processing:

[0093] The above data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.

[0094] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0095] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0096] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0097] (4) General abilities:

[0098] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0099] (5) Smart products and industry applications:

[0100] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0101] The embodiments of the present application can be applied in many fields of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities and other fields.

[0102] Specifically, the embodiments of the present application can be applied in fields that require the use of (deep) neural networks, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution and natural language processing.

[0103] The following is a brief introduction to the two application scenarios of album image classification and safe city.

[0104] Album picture categories:

[0105] When a user stores a large number of pictures on a terminal device (e.g., a mobile phone) or a cloud disk, identifying images in the album can facilitate the user or the system to categorize and manage the album, thereby improving the user experience.

[0106] By using the training method of the neural network model of the embodiment of the present application, a neural network suitable for album classification can be obtained or optimized. This neural network can then be used to classify images, thereby labeling images of different categories for easier viewing and searching by users. Furthermore, these image classification labels can also be provided to the album management system for classification management, saving users' management time, improving album management efficiency, and enhancing the user experience.

[0107] Attribute recognition in safe city scenarios:

[0108] In the safe city scenario, multiple attribute recognition is required, such as pedestrian attribute recognition and cycling attribute recognition. Deep neural networks play an important role in multiple attribute recognition with their powerful capabilities. By adopting the training method of the neural network model of the embodiment of the present application, a neural network suitable for attribute recognition in the safe city scenario can be obtained or optimized. Next, the neural network can be used to process the input road image to identify different attribute information in the road image.

[0109] The performance of neural network models is influenced by many factors, including their structure, training process, regularization methods, and hyperparameters. These factors can be controlled through human intervention or automated machine learning (AutoML). AutoML provides services based on user input of training data and target tasks.

[0110] The method of the embodiment of the present application can be used to train artificially constructed neural network models, and can also be applied to AutoML to provide corresponding services according to user needs.

[0111] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0112] (1) Neural Network

[0113] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be:

[0114]

[0115] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of , b is the bias of the neural unit.

[0116] f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of this activation function can be used as the input of the next layer. For example, the activation function can be ReLU, tanh, or sigmoid function.

[0117] A neural network is formed by connecting multiple single neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be an area consisting of several neural units.

[0118] (2) Deep Neural Networks

[0119] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0120] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0121] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0122] The activation function is usually placed after the convolutional layer or the fully connected layer. Figure 2 A fully connected layer and the activation function after the fully connected layer are shown. Figure 2 The input vector in For (x1,x2,x3), the output vector for The weight matrix W is as follows Figure 2 The weight matrix of the fully connected layer shown ( Figure 2 The superscript of the weight matrix is ​​omitted in the figure), and f represents the activation function. The input vector (x1, x2, x3) is processed by the fully connected layer and output (y1, y2), which is then processed by the activation function f to obtain the output vector

[0123] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices (a weight matrix formed by vectors from many layers) for all layers of a trained deep neural network.

[0124] (3) Convolutional Neural Networks

[0125] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as a position-independent method for extracting image information. Convolution kernels can be formalized as matrices of random size, and during CNN training, the kernels can be learned to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0126] (4) Loss function

[0127] During deep neural network training, the goal is to ensure that the output of a deep neural network is as close as possible to the desired target value. This is done by comparing the network's predictions with the desired target values ​​and then updating the weight vectors of each layer based on the difference between the two. (Of course, this initial update typically involves a process of pre-configuring parameters for each layer of the deep neural network.) For example, if the network's prediction is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the deep neural network can predict the desired target value or a value very close to it. Therefore, it is necessary to predefine how to compare the difference between the predicted and target values. This is known as the loss function or objective function, which is a key equation used to measure the difference between the predicted and target values. For example, a higher loss indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss. Generally speaking, lower loss indicates higher quality training, while higher loss indicates lower quality training. Similarly, lower loss fluctuations indicate more stable training, while higher loss fluctuations indicate more unstable training.

[0128] (5) Backpropagation algorithm

[0129] Neural networks can use the back propagation (BP) algorithm to modify the parameters of the neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters of the neural network model, thereby converging the error loss. The BP algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0130] For example, the loss value generated during each training phase of a neural network model is passed through the model layer by layer, from back to front. As it passes to each layer, the update to that layer's parameters is calculated (partial derivative operation). This update is related to the gradient. The gradient value is linearly positively correlated with the loss value.

[0131] like Figure 3 As shown, the embodiment of the present application provides a system architecture 100. Figure 3In the embodiment of the present invention, the data acquisition device 160 is used to acquire training data. For example, in the data processing method of the embodiment of the present application, if the data is image data, the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.

[0132] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0133] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input raw data and compares the output value with the target value until the difference between the value output by the training device 120 and the target value is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0134] The above-mentioned target model / rule 101 can be used to implement the data processing method of the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network model. For example, a convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0135] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 3 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) or virtual reality (VR), a vehicle terminal, etc. It can also be a server or a cloud. Figure 3 In the embodiment of the present application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: data to be processed input by the client device.

[0136] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0137] Finally, the I / O interface 112 returns the processing result, such as the processing result of the data obtained above, to the client device 140 to provide it to the user.

[0138] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0139] exist Figure 3 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0140] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0141] like Figure 3 As shown, the target model / rule 101 is obtained through training with the training device 120. The target model / rule 101 in the embodiment of the present application may be the neural network in the present application. Specifically, the neural network constructed in the embodiment of the present application may be a CNN, etc.

[0142] Figure 4 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor 50. The chip can be set as follows Figure 3 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. The method in the embodiment of the present application can be used in Figure 4 is implemented in the chip shown.

[0143] The neural network processor NPU 50 is mounted as a coprocessor on the host central processing unit (CPU), and the host CPU assigns tasks. The core of the NPU is the arithmetic circuit 503. The controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.

[0144] In some implementations, the arithmetic circuit 503 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0145] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 502 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 501 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 508.

[0146] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization (BN), local response normalization, etc.

[0147] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0148] The unified memory 506 is used to store input data and output data.

[0149] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0150] The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0151] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0152] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0153] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0154] Introduced above Figure 3 The execution device 110 or Figure 4 The chip in the embodiment of the present application can execute each step of the data processing method. Figure 3 Training device 120 or Figure 4 The chip in the embodiment of the present application can execute each step of the method for constructing the activation function. Figure 3 Training device 120 or Figure 4 The chip in can execute each step of the training method of the neural network model of the embodiment of the present application.

[0155] like Figure 5 As shown, an embodiment of the present application provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local device 301 and the local device 302 are connected to the execution device 310 via a communication network.

[0156] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and the like. The execution device 310 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the method for constructing an activation function or the method for training a neural network model according to an embodiment of the present application.

[0157] Specifically, in one implementation, the execution device 110 may perform the following process:

[0158] Get training data;

[0159] Training a neural network model based on the training data, wherein an activation function of the neural network model includes at least one piecewise function, and the piecewise function includes a plurality of trainable parameters;

[0160] A plurality of trainable parameters of at least one piecewise function are updated during a training process of training a neural network model.

[0161] By executing the above process, the device 110 can obtain a target neural network model, which can be used for image classification, image processing, speech processing or text processing, etc.

[0162] Alternatively, the device 110 can obtain a target activation function through the above process execution, and the target activation function can be applied to the neural network model to improve the performance of the neural network model.

[0163] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 310. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0164] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0165] In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the target neural network from the execution device 310, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network to perform image classification, image processing, speech processing, text processing, etc.

[0166] In another implementation, the target neural network can be directly deployed on the execution device 310. The execution device 310 obtains the data to be processed from the local device 301 and the local device 302, and uses the target neural network model to process the data to be processed.

[0167] The above-mentioned execution device 310 can also be a cloud device. In this case, the execution device 310 can be deployed in the cloud; or, the above-mentioned execution device 310 can also be a terminal device. In this case, the execution device 310 can be deployed on the user terminal side. The embodiment of the present application does not limit this.

[0168] like Figure 6 As shown, the target neural network can be deployed in the execution device 610 to process the input data provided by the user and obtain the processing result. The activation function in the target neural network is obtained by the method in the embodiment of the present application. The execution device 610 can be a terminal device or a cloud device. For example, the execution device 610 can be Figure 3 The execution device 110 in Figure 5 The execution device 310 or local device, etc.

[0169] like Figure 7 As shown, the embodiment of the present application provides a schematic diagram of the processing flow on the AutoML service platform. The AutoML service platform provides corresponding services based on the training data and target tasks provided by the user. Figure 7As shown in the figure, the AutoML service platform performs one or more search operations to find solutions that meet user needs. The AutoML service platform can perform search operations such as data augmentation strategy search, model structure search, activation function search, and hyperparameter search. Data augmentation strategy search, model structure search, and hyperparameter search are all optional operations. For example, if the user provides a model structure, there is no need to perform a model structure search.

[0170] Specifically, the activation function search can be performed using the method in the embodiment of the present application to obtain an activation function that meets the requirements. Detailed description is provided in method 800 below.

[0171] The output of the AutoML service platform is determined according to the needs of the user. In an embodiment of the present application, the output of the AutoML service platform may include a target neural network model and / or an activation function. For example, if the training data provided by the user is a sample image, and the target task is an image classification task, the AutoML service platform can output a target neural network model that can be used to perform the image classification task. For another example, if the training data provided by the user is a sample image, and the target task is an image classification task, and the user requires the output of the activation function in the target neural network model, the AutoML service platform can output the target neural network model and activation function that can be used to perform the image classification task. For another example, if the training data provided by the user is a sample image, and the target task is an image classification task, the user also provides the structure of the neural network model and requires the output of the activation function in the target neural network model, the AutoML service platform can output the activation function in the target neural network model that can be used to perform the image classification task.

[0172] Most activation functions in existing neural network models use fixed parameters or have very limited room for parameter variation. These activation functions cannot guarantee good results across all models and tasks. The rectified linear unit (ReLU) activation function is a widely used activation function in deep neural networks. However, for input features with values ​​less than 0, after ReLU processing, their output is always 0, causing some neurons to no longer activate, resulting in the corresponding parameters being permanently incapable of updating, thus affecting model performance.

[0173] The Swish function is a relatively common activation function found through reinforcement learning algorithms and can be applied to a variety of neural network models. However, the search space for activation functions using reinforcement learning algorithms is limited to predefined unary and binary functions, making it quite restrictive. Furthermore, sampling-based reinforcement learning algorithms are very inefficient and expensive to search for activation functions, making it difficult to find the optimal activation function for each task and model. The Swish function is only used as a relatively general activation function in models and is not guaranteed to be the optimal activation function for all models and tasks.

[0174] Therefore, how to obtain an activation function suitable for the neural network model and improve the performance of the neural network model has become an urgent problem to be solved.

[0175] The embodiment of the present application proposes a training method for a neural network model, which can obtain an activation function suitable for the neural network model and improve the performance of the neural network model.

[0176] The following combination Figures 8 to 13 The training method of the neural network model in the embodiment of the present application is described in detail.

[0177] Figure 8 A training method 800 for a neural network model provided in an embodiment of the present application is shown. Figure 8 The method shown can be executed by a training device for a neural network model, which can be a cloud service device or a terminal device, for example, a computer, a server, or other device with sufficient computing power to execute the training method for a neural network model, or a system composed of a cloud service device and a terminal device. Figure 3 Training equipment 120, Figure 4 Neural network processor 50 or Figure 5 Alternatively, method 800 may also be performed by a device that provides an AutoML service. For example, the device that provides the AutoML service may be a cloud service device.

[0178] For example, the method 800 may be specifically performed as follows: Figure 3 The training device 120 shown in FIG. 8 is executed, and the training data in the method 800 can be as follows: Figure 3 The training data maintained in the database 130 is shown.

[0179] In the embodiment of the present application, method 800 can also be used as a method for constructing an activation function, and the activation function in the trained neural network model is the constructed activation function. The constructed activation function can be called a target activation function. The construction of the activation function in the embodiment of the present application is achieved by adjusting the parameters of the activation function, and different parameters define different activation functions. Therefore, the method for constructing the activation function in the embodiment of the present application can also be understood as a method for searching for an activation function, and the search space of the activation function is determined by the adjustable space of the parameters of the activation function.

[0180] The method 800 includes steps S810 to S820, which are described in detail below.

[0181] S810: Obtain training data.

[0182] The type of training data is related to the task of the neural network model. For example, if the neural network model is used for an image processing task, the training data may be an image. Specifically, image processing tasks include image classification, image detection, image segmentation, or image generation. For another example, if the neural network model is used for a text processing task, the training data may be text. Specifically, text processing tasks include text recognition or text translation. For another example, if the neural network model is used for a speech processing task, the training data may be speech data. Specifically, speech processing tasks include speech recognition. The embodiments of the present application do not limit the type of training data.

[0183] Exemplarily, the training data may be pre-stored. For example, the training data may be Figure 3 The training data maintained in the database 130 is shown.

[0184] Alternatively, the training data may be input by a user. For example, method 800 is performed by a device providing an AutoML service, and the training data may be input by a user.

[0185] S820: Train a neural network model based on the training data. The activation function of the neural network model includes at least one piecewise function. The piecewise function includes multiple trainable parameters.

[0186] It should be understood that the piecewise function used as the activation function in the embodiment of the present application refers to a piecewise function including multiple trainable parameters. For the sake of simplicity, the "piecewise function including multiple trainable parameters" will be referred to as "piecewise function" in the following text.

[0187] During the training process of the neural network model, the multiple trainable parameters are updated to obtain a target neural network model.

[0188] Neural network models typically include two types of parameters. One type is trainable parameters, such as the weights of a neural network model, which can be learned and estimated from training data. The other type is hyperparameters, which are typically not learned from training data. Hyperparameters can be set manually or found through AutoML hyperparameter search.

[0189] Trainable parameters can be updated based on training data during the training process of the neural network model. Hyperparameters remain unchanged during the iterative process of neural network model training.

[0190] Specifically, during the training of the neural network model, the multiple trainable parameters are updated based on the gradients of the multiple trainable parameters.

[0191] The activation function of the neural network model includes at least one piecewise function, which can also be understood as at least one activation function in the neural network model being a piecewise function.

[0192] The activation function of the neural network model may include one or more piecewise functions. If the neural network model includes multiple piecewise functions, the parameters of the multiple piecewise functions may be the same, that is, the multiple piecewise functions may share parameters. Alternatively, the parameters of the multiple piecewise functions may be different, that is, the parameters of the multiple piecewise functions may be independent of each other. In this case, a change in the parameters of one piecewise function in the multiple piecewise functions does not affect the parameters of other piecewise functions.

[0193] For example, in a neural network model having multiple layers with piecewise functions, the parameters of the piecewise functions of different layers are different, while the parameters of the piecewise functions of the same layer are the same. In other words, each layer in the neural network model can use independent piecewise function parameters.

[0194] Furthermore, in multiple channels in the same layer of the neural network model having piecewise functions, the parameters of the piecewise functions of different channels are different. In other words, each channel in the same layer of the neural network model can use independent piecewise function parameters.

[0195] For example, all activation functions in the neural network model are piecewise functions. The activation functions in the neural network model can share the same piecewise function parameters. Alternatively, one or more layers in the neural network model with activation functions have independent piecewise function parameters. Alternatively, one or more channels in any layer with activation functions in the neural network model have independent piecewise function parameters. Independent piecewise function parameters mean that the parameters of a piecewise function can be adjusted independently without affecting other piecewise functions in the neural network model, and are not affected by changes in the parameters of other piecewise functions in the neural network model.

[0196] By using different piecewise function parameters in different channels of the same layer in the neural network model, the parameters of the piecewise function can be adjusted for different channels during the training process of the model, so that the activation functions of different channels are better adapted to the corresponding channels, further improving the performance of the model.

[0197] Exemplarily, the neural network model may be obtained by preprocessing an original neural network model. The original neural network model includes an activation function. For example, the original neural network model may be a convolutional neural network.

[0198] Preprocessing the original neural network model includes replacing at least one activation function in the original neural network model with a piecewise function. For example, all activation functions in the original neural network model are replaced with piecewise functions.

[0199] In this case, the at least one activation function in the neural network model obtained after training has been replaced by a piecewise function.

[0200] If the activation function of the original neural network model does not include a piecewise function, at least one activation function of the original neural network model can be replaced with a piecewise function to obtain the neural network model in step S820. If the activation function of the original neural network model includes a piecewise function, at least one activation function in the original neural network model that is not a piecewise function can be replaced with a piecewise function to obtain the neural network model in step S820.

[0201] Exemplarily, the neural network model or the original neural network model may be user input.

[0202] Alternatively, the neural network model or the original neural network model may be a candidate neural network model obtained by searching a sampled model structure during the model structure search process of AutoML.

[0203] A piecewise function is a function that uses different expressions for different intervals or points within its domain. The common endpoints of two adjacent intervals are called the cutoff points of the piecewise function. In other words, a piecewise function consists of multiple functions with different domains. The domain of a piecewise function is the union of the domains of the multiple piecewise functions, and the range of a piecewise function is the union of the ranges of the multiple piecewise functions.

[0204] Optionally, the at least one piecewise function is a piecewise linear function (PWL).

[0205] A piecewise linear function is a linear function with different slopes in different intervals of its domain, that is, a piecewise linear function is a linear function in every interval of its domain, and the piecewise linear function is a nonlinear function as a whole.

[0206] Using a piecewise linear function can fit various function forms, which is equivalent to providing a larger search space for the activation function search process, which can increase the upper limit of the accuracy of the searched activation function. Moreover, compared with using a piecewise nonlinear function as an activation function, using a piecewise linear function as an activation function can improve computational efficiency and reduce time overhead.

[0207] Optionally, the at least one piecewise function includes at least two dividing points, that is, the piecewise function is divided into at least three linear functions on three intervals.

[0208] A piecewise linear function can be defined by multiple parameter combinations. By setting different values ​​for the parameters, different activation functions can be obtained.

[0209] For example, a piecewise linear function includes the following parameters: N breakpoints, slopes over N+1 intervals, and function values ​​corresponding to any of the N breakpoints, where N is an integer greater than or equal to 1.

[0210] The slope on the N+1 interval refers to the slope of the piecewise linear function on the N+1 interval.

[0211] The domain of a piecewise linear function is divided into multiple intervals based on the dividing point. The piecewise linear function is a linear function in each of the multiple intervals. In other words, the piecewise linear function includes multiple linear functions in multiple intervals. The slope of the piecewise linear function in any interval among the multiple intervals is the slope of the linear function in the interval. The function value corresponding to the dividing point refers to the function value corresponding to the value of the dividing point in the piecewise linear function. In other words, in the coordinate system where the piecewise linear function is located, the value of the dividing point is used as the horizontal coordinate, and the vertical coordinate corresponds to the horizontal coordinate in the piecewise linear function.

[0212] However, the above parameter combination includes slopes in N+1 intervals. During the training process, updating the slope of any interval may affect the functions of multiple other intervals, reducing the training efficiency and the performance of the model after training.

[0213] An embodiment of the present application provides a parameter combination method, which defines the function values ​​of a piecewise linear function in the middle N-1 intervals through the function values ​​corresponding to N dividing points. The change of the function value corresponding to a dividing point only affects the functions of the two intervals adjacent to the dividing point. The impact range is small, which can improve the training efficiency and the performance of the model after training.

[0214] Optionally, at least one piecewise linear function includes one or more of the following parameters: N dividing points, the slope RK of the piecewise linear function in the interval with the largest domain, the slope LK of the piecewise linear function in the interval with the smallest domain, or the function values ​​YB corresponding to the N dividing points.

[0215] The interval with the largest domain and the interval with the smallest domain can also be called the two end intervals of the multiple intervals. The function values ​​YB corresponding to the N cutoff points can be represented as an array with N elements, each element representing a function value corresponding to a cutoff point.

[0216] Optionally, at least one piecewise function includes one or more of the following parameters: the number of dividing points N, the right boundary RB, the left boundary LB, the slope RK on the interval with the largest domain, the slope LK on the interval with the smallest domain, or the function value YB corresponding to N dividing points.

[0217] The right boundary refers to the maximum value among the N dividing points, and the left boundary refers to the minimum value among the N dividing points.

[0218] Optionally, the number of segments of the piecewise function is any value between 6 and 18.

[0219] Figure 9 A schematic diagram of a piecewise function is shown below. Figure 9 The above parameters are explained. Figure 9 The number of dividing points N in is 9, and the domain of the piecewise linear function is divided into 10 intervals based on the 9 dividing points. The interval with the largest domain among the 10 intervals is the rightmost subinterval in the coordinate system, and the subinterval with the smallest domain is the leftmost subinterval in the coordinate system. The slope RK of the piecewise linear function on the interval with the largest domain is the slope RK of the piecewise linear function on the rightmost interval. The slope RK of the piecewise linear function on the interval with the smallest domain is the slope LK of the piecewise linear function on the leftmost interval. The function values ​​YB corresponding to the N dividing points are the function values ​​of the horizontal coordinates corresponding to the N dividing points in the piecewise linear function, that is, Figure 9 The value of the vertical coordinate in the median coordinate system.

[0220] For example, the lengths of the middle intervals may be the same. The middle intervals refer to the intervals other than the intervals at both ends of the multiple intervals of the piecewise linear function.

[0221] The values ​​of the remaining N-1 dividing points can be determined based on the right boundary RB and the left boundary LB. For example, the right boundary RB and the left boundary LB are as follows Figure 9 shown.

[0222] By setting the length of the intermediate intervals to the same length, the number of parameters can be reduced and the computational efficiency can be improved.

[0223] At the same time, the above parameter combination only includes two slope values. In the subsequent parameter update process, the change of the function value corresponding to a dividing point only affects the functions of the two intervals adjacent to the dividing point. The impact range is small, which can improve the training efficiency and the performance of the model after training.

[0224] The method 800 is described below using the above parameter combination as an example.

[0225] The gradient of a parameter of at least one piecewise function is determined based on a loss function value obtained by performing a forward calculation on the training data.

[0226] During the training process of a neural network model, forward propagation of the neural network model is performed based on the training data to obtain a loss function value. Backward propagation is then performed based on this loss function value, i.e., the gradients of the model parameters are calculated and the model parameters are adjusted based on the gradients of the model parameters. For example, the model parameters may include model weights.

[0227] In step S820, the multiple trainable parameters of the piecewise function are updated using the aforementioned model parameter update method. In other words, the multiple trainable parameters of the piecewise function are treated as parameters of the neural network model, and updated parameters are obtained through backpropagation. Alternatively, this can be understood as adjusting the multiple trainable parameters of the piecewise function using the model weight update method. For example, stochastic gradient descent is used to update the multiple trainable parameters of the piecewise function.

[0228] The following combination Figure 10 The forward propagation process of the piecewise function is illustrated by way of example. The forward propagation process includes steps S1 to S7.

[0229] S1: Determine the relationship between the feature x input to the piecewise function and the right boundary RB and the left boundary LB.

[0230] If x<LB, execute step S2;

[0231] In the case of x>RB, execute step S3;

[0232] When LB≤x≤RB, execute step S4.

[0233] It should be understood that the position of the equal sign in the above relationship judgment can be set as needed. For example, the above three cases can be divided into x≤LB, LB<x≤RB, and x>RB; for another example, the above three cases can be divided into x≤LB, LB<x<RB, and x≥RB. This embodiment of the application does not limit this.

[0234] S2: The output y of the piecewise function satisfies the following formula.

[0235] y=(x-LB)*LK+YB[0];

[0236] Where YB[0] represents the function value corresponding to the first demarcation point in array YB. In the embodiment of the present application, among the demarcation points arranged in ascending order, the first demarcation point is the left boundary LB, and the Nth demarcation point is the right boundary RB. These N demarcation points, arranged in ascending order, correspond to elements YB[0], YB[1], YB[2], ..., YB[N-1] in array YB.

[0237] S3: The output y of the piecewise function satisfies the following formula.

[0238] y=(x-RB)*RK+YB[N-1];

[0239] Among them, YB[N-1] represents the function value corresponding to the Nth dividing point in the array YB, and the Nth dividing point is the right boundary RB.

[0240] S4: Calculate the length of the middle interval.

[0241] The length d of the middle interval satisfies the following formula.

[0242]

[0243] In this case, the lengths of the intermediate intervals are the same.

[0244] S5: Calculate the middle interval to which x belongs.

[0245] The number i of the middle interval to which x belongs satisfies the following formula.

[0246]

[0247] Here, floor() represents the floor function, which rounds down to the nearest integer. It is used to find the largest integer not greater than a given value. i = 0 represents the first middle interval in the middle interval, i.e., the leftmost middle interval. This continues as i increases.

[0248] S6: Calculate the left endpoint of the interval to which x belongs and the slope of the interval.

[0249] The left endpoint l of the interval to which x belongs satisfies the following formula.

[0250] l=LB+i*d;

[0251] The slope k of the interval to which x belongs satisfies the following formula.

[0252]

[0253] YB[i] represents the function value corresponding to the i+1th cutoff point, and YB[i+1] represents the function value corresponding to the i+2th cutoff point. The i+1th cutoff point is the left endpoint of the interval to which x belongs, and the i+2th cutoff point is the right endpoint of the interval to which x belongs.

[0254] S7: The output y of the piecewise function satisfies the following formula.

[0255] y=(xl)*k+YB[i];

[0256] During the training of the neural network model, each iteration can be performed according to Figure 10 The forward propagation is performed in the manner shown. The parameters of the piecewise function in the forward propagation process of each iteration are the parameters of the piecewise function adjusted after the previous iteration.

[0257] It should be understood that the above is merely illustrative, and the forward propagation process can also be performed in other ways. In addition, if the piecewise function is defined by other parameter combinations, the forward propagation process can also be adjusted accordingly based on the set parameter combinations.

[0258] The plurality of trainable parameters of the at least one piecewise function may be part or all of the parameters of the at least one piecewise function.

[0259] That is to say, some parameters in the piecewise function can be used as hyperparameters and not adjusted during the training process, while the other parameters are continuously updated as the model is trained.

[0260] Optionally, the multiple trainable parameters in at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

[0261] The number of cutoff points N is used as a hyperparameter. During the training process, the number of cutoff points is not updated, but other parameters of the piecewise function are updated.

[0262] The number of cutoff points determines the number of parameters of the piecewise function. Not updating the number of cutoff points during training can ensure that the number of parameters of the piecewise function remains unchanged, reduce the complexity of updating the piecewise function, and ensure the stability of model training.

[0263] According to the solution of the embodiment of the present application, the at least one piecewise function is used as the activation function. By adjusting the multiple trainable parameters in the at least one piecewise function, a variety of function forms can be fitted. That is, the search space for the activation function is larger, which is conducive to obtaining a more optimal activation function. At the same time, during the model training process, the multiple trainable parameters of the piecewise function are updated through parameter gradients, which reduces the operating overhead and improves the efficiency of constructing the activation function and training the model. This makes it possible to search for activation functions for different models, obtain activation functions suitable for different models, and improve the performance of the model.

[0264] The initialization form of the at least one piecewise function can be set as needed.

[0265] That is, before the first iteration, the parameters of the at least one piecewise function may be initialized as needed to obtain initial values ​​of the parameters of the at least one piecewise function.

[0266] Exemplarily, the initialization form of at least one piecewise function may be randomly set.

[0267] That is, before the first iteration, the parameters of the piecewise function can be randomly initialized.

[0268] Exemplarily, the initialization form of at least one piecewise function may be a ReLU function. For example, Figure 11 The ReLU function shown.

[0269] Before the first iteration, the parameters of the piecewise function are initialized according to the ReLU function so that the piecewise function and the ReLU function coincide on the coordinate axis.

[0270] The initial values ​​of the right boundary RB and the left boundary LB can be arbitrary. LK is set to 0, and RK is set to 1. YB is set according to the ReLU function so that the piecewise function and the ReLU function are the same, that is, the images of the two coincide on the coordinate axes.

[0271] The ReLU function is one of the most widely used activation functions. Compared to other common activation functions, models that use the ReLU function have better performance and higher training efficiency. Therefore, initializing the parameters of the piecewise function according to the ReLU function can provide a better search starting point for the activation function. Based on this, adjusting the parameters of the piecewise function can help improve model performance and training efficiency.

[0272] Alternatively, the initialization form of at least one piecewise function may be other function forms.

[0273] It should be noted that when the neural network model includes two or more piecewise functions, the initialization forms of the two or more piecewise functions may be the same or different.

[0274] Optionally, the training process for training the neural network model includes a first stage and a second stage, and the first stage is performed before the second stage. In this case, step S820 includes:

[0275] In a second stage, a plurality of trainable parameters of the at least one piecewise function are updated according to the gradients of the plurality of trainable parameters in the at least one piecewise function.

[0276] The initial values ​​of the right boundary and the left boundary of the at least one piecewise function in the second stage are determined according to the distribution of the features input to the at least one piecewise function in the first stage.

[0277] In the first stage, the distribution of features input to the at least one piecewise function is statistically analyzed, and then the values ​​of the right boundary and the left boundary of the at least one piecewise function are reset based on the distribution of the features statistically analyzed in the first stage. The reset right boundary value is used as the initial value of the right boundary of the at least one piecewise function in the second stage, and the reset left boundary value is used as the initial value of the left boundary of the at least one piecewise function in the second stage. The initial values ​​of other parameters of the at least one piecewise function in the second stage can be the values ​​of other parameters of the at least one piecewise function at the end of the first stage. Then, the second stage of training is performed, and the multiple trainable parameters of the at least one piecewise function are updated based on the gradients of the multiple trainable parameters during the iterative process of the second stage.

[0278] For a piecewise function among the at least one piecewise function, initial values ​​of its right boundary and left boundary in the second stage are determined according to the distribution of the features input to the piecewise function in the first stage.

[0279] In the first stage, a plurality of trainable parameters of the at least one piecewise function remain unchanged.

[0280] During the iterative process of the first stage, the parameters of the at least one piecewise function are fixed, and the distribution of the features input to the at least one piecewise function is counted.

[0281] In the first stage, the parameters of the piecewise function are fixed to avoid the influence of the parameter changes of the piecewise function on the distribution of the input features. This can obtain a more accurate distribution of the input features, which is conducive to the subsequent adjustment of the parameters of the piecewise function.

[0282] The division method of the first stage and the second stage can be set as needed.

[0283] Exemplarily, the first stage and the second stage are set according to the number of epochs for model training.

[0284] An epoch can be understood as the process of training all the training data in the training set once.

[0285] The first phase can include fewer epochs. For example, if the model training includes 100 epochs in total, the first phase includes the first 5 epochs and the second phase includes 95 epochs.

[0286] Alternatively, the first stage and the second stage are set according to the distribution of features input to the at least one piecewise function during each iteration of the first stage.

[0287] Specifically, the first stage may be terminated and the second stage may be initiated after the distribution of the features input to the at least one piecewise function becomes stable. For example, the first stage may be terminated and the second stage may be initiated when the change in the distribution of the features input to the at least one piecewise function is less than a set threshold.

[0288] Optionally, the distribution of the features input to the at least one piecewise function in the first stage is represented by a predicted mean value of the features and a predicted standard deviation of the features obtained after the last iteration of the first stage. The predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0289] For example, the features input to the at least one piecewise function in the first stage may be considered to obey a normal distribution. In this case, the distribution of the features of the at least one piecewise function in the first stage may be represented by a normal distribution, thereby determining the initial values ​​of the right boundary and the initial values ​​of the left boundary of the parameters of the at least one piecewise function in the second stage.

[0290] Specifically, the normal distribution of the features input to the at least one piecewise function may be determined based on the predicted mean and the predicted standard deviation of the features input to the at least one piecewise function in the first stage.

[0291] The predicted mean and standard deviation of a feature are determined using a sliding average method. That is, in each iteration of the first stage, the predicted mean of the feature after the current iteration is determined based on the predicted mean of the feature and the average value of the feature obtained after the previous iteration. In each iteration of the first stage, the predicted standard deviation of the feature after the current iteration is determined based on the predicted standard deviation of the feature obtained after the previous iteration and the standard deviation of the feature.

[0292] That is, in the first stage, the predicted mean and predicted standard deviation of the feature input to the at least one piecewise function are updated during each iteration until the end of the first stage. The normal distribution of the feature is determined based on the predicted mean and predicted standard deviation of the feature obtained after the last iteration of the first stage, that is, the initial values ​​of the right and left boundaries of the at least one piecewise function in the second stage are determined.

[0293] For example, for any piecewise function, the predicted mean R of the feature mean and the predicted standard deviation R of the feature std Satisfies the following formula.

[0294]

[0295]

[0296] Among them, R mean_j represents the predicted average value of the feature after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, R std_j represents the predicted standard deviation of the feature after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, R mean_0 represents the initial value of the predicted mean of the feature, R std_0 Represents the initial value of the predicted standard deviation of the feature. The above formula represents the predicted mean value R of the feature. mean_0 and the predicted standard deviation R of the feature std_0 Initialized to 0. mean(x) represents the mean value of the calculated feature, std(x) represents the standard deviation of the calculated feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0297] a is a positive number less than 1, and b is a positive number less than 1. For example, a is 0.9 and b is 0.9.

[0298] The predicted mean and standard deviation of the feature obtained after the last iteration of the first stage are used as the mean and variance of the normal distribution that the feature follows. The right and left bounds of the piecewise function are set according to the 3 sigma principle. The reset right and left bounds of the piecewise function serve as the initial values ​​of the right and left bounds of the piecewise function in the second stage.

[0299] Optionally, initial values ​​of the right boundary RB and the left boundary LB in at least one piecewise function in the second stage satisfy the following formula.

[0300] RB=R mean +c*Rstd ;

[0301] LB=R mean -c*R std ;

[0302] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R std Represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents a parameter. c is a positive number, for example, c is 3.

[0303] For variables that follow a normal distribution, more than 68% of the variables are within 1 standard deviation from the mean, more than 95% of the variables are within 2 standard deviations from the mean, and more than 99% of the variables are within 3 standard deviations from the mean. Setting c to 3 can make the features input to the piecewise function fall as much as possible between the left and right boundaries of the activation function.

[0304] In the early stage of model training, the distribution of input features varies greatly. In an embodiment of the present application, the values ​​of the left and right boundaries of the activation function are adjusted based on the distribution of input features obtained by statistics in the first stage. In this way, the left and right boundaries can match the actual distribution of input features, so that the input features fall between the left and right boundaries as much as possible. In this way, more parameters in the activation function can be involved in the training process, and multiple segments in the piecewise function can be effectively utilized. In other words, the parameters of the activation function can be effectively adjusted using training data, thereby stabilizing the training process, improving the accuracy of the activation function, and thus improving the performance of the model and the training efficiency.

[0305] After model training is completed, an activation function and a trained model can be obtained. In other words, method 800 can be used to obtain both an activation function and a trained model. Therefore, method 800 can be used as a method for constructing an activation function to obtain a constructed activation function, and can also be used as a method for training a neural network model to obtain a trained neural network model.

[0306] In one implementation, the activation function obtained using method 800 matches the trained model and can improve the performance of the model.

[0307] In another implementation, the activation function obtained using method 800 can be applied to other models, for example, to a model with a similar model structure to that in method 800. This can further improve the training efficiency of the model while ensuring the performance of the model.

[0308] The trained neural network model can be used to perform the target task. For example, the target task can be an image processing task, such as object detection, image segmentation, instance segmentation, image denoising, image super-resolution, etc. Alternatively, the target task can be a speech processing task, such as speech recognition, etc. Alternatively, the target task can be a text processing task, such as text recognition or text translation, etc.

[0309] In one implementation, the neural network model in step S820 is obtained by preprocessing the original neural network model, which includes an activation function.

[0310] Specifically, at least one activation function in the original neural network model can be replaced with a piecewise function, and the replaced neural network model can be trained using method 800 to obtain a constructed activation function and a trained neural network model.

[0311] For example, the original neural network model can be a neural network model input by a user. For example, the model can be a manually constructed neural network model or a neural network model obtained by AutoML. For another example, the model can be an existing model, for example, a convolutional neural network model or a ResNet18 model.

[0312] Alternatively, the neural network model to be processed can be a candidate neural network model searched during the execution of AutoML.

[0313] During the execution of AutoML, a sampling-based model structure search is performed to obtain candidate neural network models. The candidate neural network models are then trained and repeated until a neural network model that meets the requirements is obtained. The original neural network model can be one of the candidate neural network models.

[0314] In another implementation, the activation function of the neural network model in step S820 includes at least one piecewise function, that is, during the construction of the neural network model, at least one activation function is set as a piecewise function.

[0315] Exemplarily, the model may be a neural network model constructed artificially or a neural network model obtained by AutoML.

[0316] Furthermore, the number of cutoff points can be used as a hyperparameter, and the number of cutoff points can be updated by searching for the hyperparameter during the execution of AutoML.

[0317] Figure 12 A training method for a neural network model is shown. Figure 12The method can be regarded as a specific implementation of the method 800. For a specific description, please refer to the aforementioned method 800. In order to avoid repetition, Figure 12 The method may be performed by the apparatus 1200 capable of performing neural network model training.

[0318] Method 1200 includes steps S1210 to S1250.

[0319] S1210: Replace the activation function in the original neural network model with at least one piecewise function.

[0320] For example, some or all activation functions in the original neural network model can be replaced with at least one piecewise function. If the at least one piecewise function includes two or more piecewise functions, the parameters of the multiple piecewise functions can be partially or completely independent of each other, or the multiple piecewise functions can share a set of parameters.

[0321] For example, the neural network model to be processed may be ResNet18. The piecewise function may be as follows Figure 9 The PWL form shown. The parameters of the piecewise function in each channel of the replaced neural network model are independent.

[0322] Exemplarily, the original neural network model may be provided by a user.

[0323] S1220: Initialize the at least one piecewise function as a ReLU function.

[0324] Initialize the parameters of the at least one piecewise function so that the at least one piecewise function satisfies the form of the ReLU function, and obtain the initial values ​​of the parameters of the at least one piecewise function. For example, the initialized piecewise function can be as follows Figure 10 shown.

[0325] The number of cutoff points is a hyperparameter and does not participate in parameter updates during training. The number of cutoff points can be 5-17. In other words, the number of segments of the piecewise function can be any value between 6 and 18. For example, the number of cutoff points can be 9.

[0326] The initial value of the right boundary RB and the initial value of the left boundary LB can be any value. For example, the initial value of the right boundary is 1, and the initial value of the left boundary is -1.

[0327] LK is set to 0 and RK is set to 1.

[0328] Set YB according to the ReLU function so that the piecewise function and the ReLU function are the same, that is, their images coincide on the coordinate axis.

[0329] In step S1220 , initializing all piecewise functions to ReLU functions is merely taken as an example.

[0330] Alternatively, some of the at least one piecewise function may be initialized as ReLU functions. Detailed descriptions can be found in step S820 of the aforementioned method 800 and will not be repeated here.

[0331] S1230: Execute the first stage of training. In the first stage, the parameters of the at least one piecewise function remain unchanged, and statistics are collected on the distribution of features of each piecewise function input into the at least one piecewise function.

[0332] That is to say, in the first stage, the training of the model weights is performed only based on the training data, and the parameters of the at least one piecewise function remain in an initialized state, that is, in the first stage, the parameters of the at least one piecewise function are always fixed to make the at least one piecewise function like the parameters of the ReLU function.

[0333] For example, the training data may be provided by a user.

[0334] For example, the distribution of the features input to each piecewise function satisfies the following formula.

[0335]

[0336]

[0337] Specifically, the predicted mean R of the feature mean_0 and the predicted standard deviation R of the feature std_0 Initialized to 0, the predicted mean and predicted standard deviation are updated in each iteration of the first stage. The predicted standard deviation and predicted standard deviation obtained after the last iteration of the first stage are used to indicate the distribution of the features input into the piecewise function.

[0338] Exemplarily, the training process includes 100 epochs, the first stage may include 5 epochs, and the second stage includes 95 epochs.

[0339] It should be understood that step S1230 is merely described by taking the at least one piecewise function as an example of being independent of each other, and does not limit the solution of the embodiment of the present application.

[0340] S1240: Adjust the right boundary and the left boundary of each piecewise function based on the distribution of the features of each piecewise function input into the at least one piecewise function.

[0341] Exemplarily, the right and left bounds of each piecewise function satisfy the formula.

[0342] RB=R mean +3*R std ;

[0343] LB=R mean -3*R std ;

[0344] The adjusted right and left boundaries of each piecewise function are the initial values ​​of the right and left boundaries of each piecewise function in the second stage. The initial values ​​of other parameters of each piecewise function in the second stage are the values ​​of the parameters of each piecewise function after initialization in step S1220.

[0345] S1250: Execute the second stage of training. In the second stage, the multiple trainable parameters of the at least one piecewise function are updated based on the gradients of the multiple trainable parameters of the at least one piecewise function.

[0346] That is, in the second stage, the weights of the model and the multiple trainable parameters of the at least one piecewise function are updated, for example, using a stochastic gradient descent method.

[0347] The multiple trainable parameters of the at least one piecewise function in step S1250 refer to the parameters of the at least one piecewise function excluding the number of cutoff points. Method 1200 uses the number of cutoff points as a hyperparameter to update the remaining parameters of the piecewise function as an example. Other parameters of the piecewise function can also be set as hyperparameters to update the remaining parameters of the piecewise function.

[0348] After the second stage is completed, the weights of the trained model and the corresponding activation function parameters are obtained. The trained model can then be deployed and used for inference tasks.

[0349] The completion of the second stage can also be understood as the end of the activation function search process, and the parameters of the constructed activation function are obtained.

[0350] Table 1 shows the comparative results of multiple models using the PWL function in the embodiment of the present application as the activation function and using other functions (ReLU and Swish) as the activation function on the Imagenet image classification task.

[0351] Table 1

[0352]

[0353] Here, model represents the neural network model selected for performing image classification tasks, including the MobileNetV3 model, ResNet18 model, and EfficientNet-B0 model.

[0354] Search overhead refers to the resource overhead consumed in training the model, that is, the resource overhead consumed when obtaining a model that can be used for reasoning. Search overhead comparison refers to the resource overhead consumed for a single training session compared to a training method that does not require searching for an activation function. The search overhead of Swish is more than 300 times the resource overhead consumed for a single training session based on a training method that does not require searching for an activation function. The search overhead of the PWL of this application is only about 1.2 times the resource overhead consumed for a single training session based on a training method that does not require searching for an activation function, which means that the construction process of the PWL of this application does not require a large amount of resource overhead. In addition, the units of the values ​​in Table 1 are percentages, which are used to represent the top-1 accuracy in image classification tasks. It can be seen that among different models, the accuracy of the model using PWL in classification tasks has been greatly improved compared to the models using the other two activation functions. The PWL function has high search efficiency and can significantly improve the accuracy of the model.

[0355] Figure 13 (a) shows the form of the piecewise function in ResNet18 obtained based on method 1300. Figure 13 (b) shows the form of existing activation functions, such as ReLU activation function, Swish activation function, Softplus activation function and Mish activation function. Figure 13 It can be seen that compared with the existing activation function, the PWL function has more diverse forms. The solution of the embodiment of the present application can adaptively obtain the activation function suitable for the model according to different models.

[0356] Figure 14 The schematic flow chart of the data processing method 1400 provided in the embodiment of the present application is shown. The method can be executed by a device or equipment capable of performing data processing. For example, the device can be a cloud service device or a terminal device, such as a computer, a server, or other device with sufficient computing power to execute the data processing method, or a system composed of a cloud service device and a terminal device. For example, the method 1400 can be performed by Figure 3 The execution device 110 in Figure 4 Neural network processor 50 or Figure 5 The execution device 310 or local device executes.

[0357] For example, the method 1400 may be specifically performed as follows: Figure 3 The execution device 110 shown in FIG. 14 is executed, and the data to be processed in the method 1400 may be as follows: Figure 3 Input data is provided by client device 140 as shown.

[0358] Figure 14 The target neural network model used in the data processing method 1400 can be the above Figure 8 Method 1400 includes steps S1410 to S1420. The specific implementation of method 1400 can refer to the aforementioned method 800. To avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 1400.

[0359] S1410: Obtain data to be processed. The data to be processed includes image data, voice data, or text data.

[0360] The type of data to be processed is related to the task of the neural network model. For example, if the neural network model is used for an image processing task, the data to be processed may be an image. Specifically, image processing tasks include image classification, image detection, image segmentation, image recognition, or image generation. For another example, if the neural network model is used for a text processing task, the data to be processed may be text. Specifically, text processing tasks include text recognition or text translation. For another example, if the neural network model is used for a speech processing task, the data to be processed may be speech data. Specifically, speech processing tasks include speech recognition, etc. The embodiment of the present application does not limit the type of data to be processed.

[0361] Exemplarily, the data to be processed is an image, and the image to be processed may be an image captured by a terminal device (or other apparatuses or devices such as a computer or server) through a camera, or the image to be processed may also be an image obtained from inside the terminal device (or other apparatuses or devices such as a computer or server) (for example, an image stored in the photo album of the terminal device, or an image obtained by the terminal device from the cloud). The embodiments of the present application are not limited to this.

[0362] S1420, using the target neural network model to process the data to be processed to obtain a processing result of the data to be processed.

[0363] Among them, the target neural network model is obtained by training the neural network model based on the training data, the activation function of the neural network model includes at least one piecewise function, and the activation function of the target neural network model includes at least one target piecewise function. The target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model.

[0364] According to the solution of the embodiment of the present application, at least one piecewise function is used as the activation function. By adjusting the multiple trainable parameters of the at least one piecewise function, a variety of function forms can be fitted. That is, the search space for the activation function is larger, which is conducive to obtaining a more optimal activation function. At the same time, during the model training process, the multiple trainable parameters of the piecewise function are updated through parameter gradients, which reduces the operating overhead and improves the efficiency of constructing the activation function and training the model. This makes it possible to search for the activation function for a specific model, obtain a suitable activation function, and improve the performance of the target neural network model.

[0365] Optionally, at least one piecewise function is a piecewise linear function, and the parameters of at least one piecewise function include one or more of the following: the number of dividing points, the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0366] Optionally, the training process for training the neural network model includes a first stage and a second stage, the first stage is executed before the second stage, and the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model, including: the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function according to the gradients of the multiple trainable parameters in the piecewise function in the second stage, wherein the initial values ​​of the right boundary and the left boundary in the piecewise function in the second stage are determined according to the distribution of features input to the piecewise function in the first stage.

[0367] Optionally, in the first stage, a plurality of trainable parameters in at least one piecewise function remain unchanged.

[0368] Optionally, the distribution of features input to the piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0369] Optionally, the predicted mean of the feature and the predicted standard deviation of the feature satisfy the following formula:

[0370]

[0371] Among them, R mean_j represents the predicted average value of the feature obtained after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, R std_j Represents the predicted standard deviation of the feature obtained after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, Rmean_0 represents the initial value of the predicted mean of the feature, R std_0 represents the initial value of the predicted standard deviation of the feature, R mean_0 =0, R std_0 =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0372] Optionally, an initial value of the right boundary RB in at least one piecewise function in the second stage satisfies the following formula:

[0373] RB=R mean +c*R std ;

[0374] The initial value of the left boundary LB in at least one piecewise function in the second stage satisfies the following formula:

[0375] LB=R mean -c*R std ;

[0376] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R std represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

[0377] Optionally, the multiple trainable parameters in at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

[0378] Optionally, the number of segments of the piecewise function is any value between 6 and 18.

[0379] The following combination Figures 15 to 18 The apparatus of the embodiment of the present application is described. It should be understood that the apparatus described below can execute the method of the embodiment of the present application, and in order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the apparatus of the embodiment of the present application.

[0380] Figure 15 It is a schematic block diagram of a training device for a neural network model according to an embodiment of the present application. Figure 15 The training device 3000 of the neural network model shown includes an acquisition unit 3010 and a processing unit 3020.

[0381] The acquisition unit 3010 and the processing unit 3020 can be used to execute the training method of the neural network model of the embodiment of the present application, specifically, can be used to execute method 800.

[0382] The acquisition unit 3010 is used to acquire training data.

[0383] The processing unit 3020 is used to train the neural network model based on the training data, where the activation function of the neural network model includes at least one piecewise function, and the piecewise function includes multiple trainable parameters; during the training process of the neural network model, the multiple trainable parameters of the at least one piecewise function are updated to obtain the target neural network model.

[0384] Optionally, as an embodiment, at least one piecewise function is a piecewise linear function, and the parameters of at least one piecewise function include one or more of the following: the number of dividing points, the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0385] Optionally, as an embodiment, the training process for training the neural network model includes a first stage and a second stage, the first stage is executed before the second stage, and the processing unit 3020 is specifically used to: in the second stage, update multiple trainable parameters of at least one piecewise function according to the gradients of multiple trainable parameters in at least one piecewise function; wherein the initial values ​​of the right boundary and the left boundary in the at least one piecewise function in the second stage are determined according to the distribution of features input to the at least one piecewise function in the first stage.

[0386] Optionally, as an embodiment, in the first stage, multiple trainable parameters in at least one piecewise function remain unchanged.

[0387] Optionally, as an embodiment, the distribution of features input into at least one piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0388] Optionally, as an embodiment, the predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula:

[0389]

[0390] Among them, R mean_j represents the predicted average value of the feature obtained after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, R std_j Represents the predicted standard deviation of the feature obtained after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, R mean_0represents the initial value of the predicted mean of the feature, R std_0 represents the initial value of the predicted standard deviation of the feature, R mean_0 =0, R std_0 =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0391] Optionally, as an embodiment, an initial value of the right boundary RB in at least one piecewise function in the second stage satisfies the following formula:

[0392] RB=R mean +c*R std ;

[0393] The initial value of the left boundary LB in at least one piecewise function in the second stage satisfies the following formula:

[0394] LB=R mean -c*R std ;

[0395] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R std represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

[0396] Optionally, as an embodiment, the multiple trainable parameters in at least one piecewise function include: a right boundary, a left boundary, a slope on the interval with the largest domain, a slope on the interval with the smallest domain, or a function value corresponding to a dividing point.

[0397] Optionally, as an embodiment, the number of segments of the piecewise function is any value from 6 to 18.

[0398] Figure 16 It is a schematic block diagram of the data processing device 4000 provided in an embodiment of the present application. Figure 16 The data processing device 4000 shown includes an acquisition unit 4010 and a processing unit 4020 .

[0399] The acquisition unit 4010 and the processing unit 4020 can be used to execute the data processing method of the embodiment of the present application, for example, can be used to execute method 1400.

[0400] The acquisition unit 4010 is used to acquire data to be processed.

[0401] The processing unit 4020 is used to process the data to be processed using the target neural network model to obtain a processing result of the data to be processed.

[0402] Among them, the target neural network model is obtained by training the neural network model based on the training data, the activation function of the target neural network model includes at least one target piecewise function, the activation function of the neural network model includes at least one piecewise function, and the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model.

[0403] Optionally, as an embodiment, at least one piecewise function is a piecewise linear function, and the parameters of at least one piecewise function include one or more of the following: the number of dividing points, the right boundary, the left boundary, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

[0404] Optionally, as an embodiment, the training process for training the neural network model includes a first stage and a second stage, the first stage is executed before the second stage, and the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model, including: the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function according to the gradients of the multiple trainable parameters in the piecewise function in the second stage, wherein the initial values ​​of the right boundary and the left boundary in the piecewise function in the second stage are determined according to the distribution of features input to the piecewise function in the first stage.

[0405] Optionally, as an embodiment, in the first stage, multiple trainable parameters in at least one piecewise function remain unchanged.

[0406] Optionally, as an embodiment, the distribution of features input to the piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

[0407] Optionally, as an embodiment, the predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula:

[0408]

[0409] Among them, R mean_j represents the predicted average value of the feature obtained after the jth iteration, R mean_j+1 represents the predicted average value of the feature obtained after the j+1th iteration, R std_j Represents the predicted standard deviation of the feature obtained after the jth iteration, R std_j+1 Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0. When j=0, R mean_0 represents the initial value of the predicted mean of the feature, Rstd_0 represents the initial value of the predicted standard deviation of the feature, R mean_0 =0, R std_0 =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents R mean_j The weight parameter, b represents R std_j The weight parameter of .

[0410] Optionally, as an embodiment, an initial value of the right boundary RB in at least one piecewise function in the second stage satisfies the following formula:

[0411] RB=R mean +c*R std ;

[0412] The initial value of the left boundary LB in at least one piecewise function in the second stage satisfies the following formula:

[0413] LB=R mean -c*R std ;

[0414] Among them, R mean represents the predicted average value of the features obtained after the last iteration of the first stage; R std represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

[0415] Optionally, as an embodiment, the multiple trainable parameters in at least one piecewise function include: a right boundary, a left boundary, a slope on the interval with the largest domain, a slope on the interval with the smallest domain, or a function value corresponding to a dividing point.

[0416] Optionally, as an embodiment, the number of segments of the piecewise function is any value from 6 to 18.

[0417] It should be noted that the training apparatus 3000 and the apparatus 4000 are implemented in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0418] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0419] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0420] Figure 17 This is a schematic diagram of the hardware structure of the training device for the neural network model provided in an embodiment of the present application. Figure 17 The neural network model training apparatus 5000 shown (the apparatus 5000 may be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, the processor 5002, and the communication interface 5003 are connected to each other via the bus 5004.

[0421] The memory 5001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 may store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is used to execute the various steps of the training method of the neural network model of the embodiment of the present application. Specifically, the processor 5002 may execute the above Figure 8 Step S820 in the method shown.

[0422] The processor 5002 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the training method of the neural network model of the method embodiment of the present application.

[0423] The processor 5002 may also be an integrated circuit chip having signal processing capabilities, for example, Figure 4 During implementation, each step of the neural network model training method of the present application can be completed by hardware integrated logic circuits in the processor 5002 or software instructions.

[0424] The above-mentioned processor 5002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 5001, and the processor 5002 reads the information in the memory 5001 and completes it in combination with its hardware. Figure 15 The functions required to be performed by the units included in the training device shown, or the functions required to perform the method embodiments of the present application Figure 8 The training method of the neural network model shown.

[0425] The communication interface 5003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 5000 and other devices or a communication network. For example, training data can be obtained through the communication interface 5003.

[0426] The bus 5004 may include a path for transmitting information between various components of the device 5000 (eg, the memory 5001 , the processor 5002 , and the communication interface 5003 ).

[0427] Figure 18 Schematic diagram of the hardware structure of the data processing device according to the embodiment of the present application. Figure 18 The data processing device 6000 shown includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, the processor 6002, and the communication interface 6003 are communicatively connected to each other via the bus 6004.

[0428] The memory 6001 may be a ROM, a static storage device, or a RAM. The memory 6001 may store a program. When the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute the various steps of the data processing method of the embodiment of the present application. Specifically, the processor 6002 may execute the above Figure 14 Step S1420 in the method shown

[0429] The processor 6002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the data processing device of the embodiment of the present application, or to execute the data processing method of the method embodiment of the present application.

[0430] The processor 6002 may also be an integrated circuit chip having signal processing capabilities, for example, Figure 4 During implementation, each step of the data processing method of the embodiment of the present application can be completed by hardware integrated logic circuits in the processor 6002 or by software instructions.

[0431] The processor 6002 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor, or it may be any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module may be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory 6001. The processor 6002 reads the information in the memory 6001 and, in conjunction with its hardware, performs the functions required to be performed by the units included in the data processing device of the embodiments of this application, or executes the data processing method of the method embodiments of this application.

[0432] The communication interface 6003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 6000 and other devices or a communication network. For example, data to be processed can be obtained through the communication interface 6003.

[0433] The bus 6004 may include a path for transmitting information between various components of the device 6000 (eg, the memory 6001 , the processor 6002 , and the communication interface 6003 ).

[0434] It should be noted that although the above-mentioned apparatus 5000 and apparatus 6000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include only the devices necessary to implement the embodiments of the present application, and do not necessarily include Figure 17 and Figure 18 All devices shown in .

[0435] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0436] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0437] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0438] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0439] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0440] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0441] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0442] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0443] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0444] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0445] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0446] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0447] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A training method for a neural network model, characterized in that: include: Acquire training data, where the training data includes: image data, voice data, or text data; Training a neural network model based on the training data, wherein an activation function of the neural network model includes at least one piecewise function, and the piecewise function includes a plurality of trainable parameters; updating a plurality of trainable parameters of the at least one piecewise function during a training process for training the neural network model to obtain a target neural network model, wherein the parameters of the at least one piecewise function include a left boundary and a right boundary, and the training process for training the neural network model includes a first stage and a second stage, wherein the first stage is performed before the second stage; and The updating of the plurality of trainable parameters of the at least one piecewise function during the training process of the neural network model comprises: In the second stage, updating the plurality of trainable parameters of the at least one piecewise function according to the gradients of the plurality of trainable parameters in the at least one piecewise function; The initial values ​​of the right boundary and the left boundary of the at least one piecewise function in the second stage are determined according to the distribution of the features input to the at least one piecewise function in the first stage.

2. The method according to claim 1, characterized in that The at least one piecewise function is a piecewise linear function, and the parameters of the at least one piecewise function also include one or more of the following: the number of dividing points, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

3. The method according to claim 1 or 2, characterized in that In the first stage, a plurality of trainable parameters in the at least one piecewise function remain unchanged.

4. The method according to claim 1 or 2, characterized in that The distribution of the features input to the at least one piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

5. The method according to claim 4, characterized in that The predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula: ； in, represents the predicted average value of the feature obtained after the j-th iteration, represents the predicted average value of the feature obtained after the j+1th iteration, represents the predicted standard deviation of the feature obtained after the jth iteration, Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0, when j=0, represents the initial value of the predicted mean of the feature, represents the initial value of the predicted standard deviation of the feature, =0, =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents The weight parameter, b represents The weight parameter of .

6. The method according to claim 4, characterized in that The initial value of the right boundary RB in the at least one piecewise function in the second stage satisfies the following formula: ； The initial value of the left boundary LB of the at least one piecewise function in the second stage satisfies the following formula: ； in, represents the predicted average value of the features obtained after the last iteration of the first stage; represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

7. The method according to claim 2, characterized in that The multiple trainable parameters in the at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

8. The method according to claim 1 or 2, characterized in that The number of segments of the piecewise function is any value from 6 to 18.

9. A data processing method, characterized in that: include: Acquiring data to be processed, the data including: image data, voice data or text data; Processing the data to be processed using a target neural network model to obtain a processing result of the data to be processed; The target neural network model is obtained by training a neural network model based on training data, the activation function of the neural network model includes at least one piecewise function, the activation function of the target neural network model includes at least one target piecewise function, the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model, the parameters of the at least one piecewise function include a right boundary and a left boundary, the training process for training the neural network model includes a first stage and a second stage, the first stage is performed before the second stage, and The target piecewise function is obtained by updating a plurality of trainable parameters of the piecewise function during the training of the neural network model, including: The target piecewise function is obtained by updating multiple trainable parameters of the piecewise function according to the gradients of the multiple trainable parameters in the piecewise function in the second stage, wherein the initial values ​​of the right boundary and the left boundary in the piecewise function in the second stage are determined according to the distribution of features input to the piecewise function in the first stage.

10. The method according to claim 9, characterized in that The at least one piecewise function is a piecewise linear function, and the parameters of the at least one piecewise function also include one or more of the following: the number of dividing points, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

11. The method according to claim 9 or 10, characterized in that In the first stage, a plurality of trainable parameters in the at least one piecewise function remain unchanged.

12. The method according to claim 9 or 10, characterized in that The distribution of the features input to the piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

13. The method according to claim 12, characterized in that The predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula: ； in, represents the predicted average value of the feature obtained after the j-th iteration, represents the predicted average value of the feature obtained after the j+1th iteration, represents the predicted standard deviation of the feature obtained after the jth iteration, Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0, when j=0, represents the initial value of the predicted mean of the feature, represents the initial value of the predicted standard deviation of the feature, =0, =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents The weight parameter, b represents The weight parameter of .

14. The method according to claim 12, characterized in that The initial value of the right boundary RB in the at least one piecewise function in the second stage satisfies the following formula: ； The initial value of the left boundary LB of the at least one piecewise function in the second stage satisfies the following formula: ； in, represents the predicted average value of the features obtained after the last iteration of the first stage; represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

15. The method according to claim 10, characterized in that The multiple trainable parameters in the at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

16. The method according to claim 9 or 10, characterized in that The number of segments of the piecewise function is any value from 6 to 18.

17. A training device for a neural network model, characterized in that: include: An acquisition unit, configured to acquire training data, wherein the training data includes image data, voice data, or text data; A processing unit for: Training a neural network model based on the training data, wherein an activation function of the neural network model includes at least one piecewise function, and the piecewise function includes a plurality of trainable parameters; updating a plurality of trainable parameters of the at least one piecewise function during a training process for training the neural network model to obtain a target neural network model, wherein the parameters of the at least one piecewise function include a right boundary and a left boundary, and the training process for training the neural network model includes a first stage and a second stage, wherein the first stage is performed before the second stage; and The processing unit is specifically configured to: In the second stage, updating the plurality of trainable parameters of the at least one piecewise function according to the gradients of the plurality of trainable parameters in the at least one piecewise function; The initial values ​​of the right boundary and the left boundary of the at least one piecewise function in the second stage are determined according to the distribution of the features input to the at least one piecewise function in the first stage.

18. The training device according to claim 17, characterized in that The at least one piecewise function is a piecewise linear function, and the parameters of the at least one piecewise function also include one or more of the following: the number of dividing points, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

19. The training device according to claim 17 or 18, characterized in that In the first stage, a plurality of trainable parameters in the at least one piecewise function remain unchanged.

20. The training device according to claim 17 or 18, characterized in that The distribution of the features input to the at least one piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

21. The training device according to claim 20, characterized in that The predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula: ； in, represents the predicted average value of the feature obtained after the j-th iteration, represents the predicted average value of the feature obtained after the j+1th iteration, represents the predicted standard deviation of the feature obtained after the jth iteration, Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0, when j=0, represents the initial value of the predicted mean of the feature, represents the initial value of the predicted standard deviation of the feature, =0, =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents The weight parameter, b represents The weight parameter of .

22. The training device according to claim 20, characterized in that The initial value of the right boundary RB in the at least one piecewise function in the second stage satisfies the following formula: ； The initial value of the left boundary LB of the at least one piecewise function in the second stage satisfies the following formula: ； in, represents the predicted average value of the features obtained after the last iteration of the first stage; represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

23. The training device according to claim 18, characterized in that The multiple trainable parameters in the at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

24. The training device according to claim 17 or 18, characterized in that The number of segments of the piecewise function is any value from 6 to 18.

25. A data processing device, characterized in that: include: An acquisition unit, configured to acquire data to be processed, wherein the data includes image data, voice data or text data; A processing unit for: Processing the data to be processed using a target neural network model to obtain a processing result of the data to be processed; The target neural network model is obtained by training a neural network model based on training data, the activation function of the neural network model includes at least one piecewise function, the activation function of the target neural network model includes at least one target piecewise function, the target piecewise function is obtained by updating multiple trainable parameters of the piecewise function during the training of the neural network model, the parameters of the at least one piecewise function include a right boundary and a left boundary, the training process for training the neural network model includes a first stage and a second stage, the first stage is performed before the second stage, and The target piecewise function is obtained by updating a plurality of trainable parameters of the piecewise function during the training of the neural network model, including: The target piecewise function is obtained by updating multiple trainable parameters of the piecewise function according to the gradients of the multiple trainable parameters in the piecewise function in the second stage, wherein the initial values ​​of the right boundary and the left boundary in the piecewise function in the second stage are determined according to the distribution of features input to the piecewise function in the first stage.

26. The device according to claim 25, characterized in that The at least one piecewise function is a piecewise linear function, and the parameters of the at least one piecewise function also include one or more of the following: the number of dividing points, the slope on the interval with the largest domain, the slope on the interval with the smallest domain, or the function value corresponding to the dividing point.

27. The device according to claim 25 or 26, characterized in that In the first stage, a plurality of trainable parameters in the at least one piecewise function remain unchanged.

28. The device according to claim 25 or 26, characterized in that The distribution of the features input to the piecewise function in the first stage is represented by the predicted mean value of the features and the predicted standard deviation of the features obtained after the last iteration of the first stage, and the predicted mean value of the features and the predicted standard deviation of the features are determined by a sliding average method.

29. The device according to claim 28, characterized in that The predicted mean value of the feature and the predicted standard deviation of the feature satisfy the following formula: ； in, represents the predicted average value of the feature obtained after the j-th iteration, represents the predicted average value of the feature obtained after the j+1th iteration, represents the predicted standard deviation of the feature obtained after the jth iteration, Represents the predicted standard deviation of the feature obtained after the j+1th iteration, j is an integer greater than or equal to 0, when j=0, represents the initial value of the predicted mean of the feature, represents the initial value of the predicted standard deviation of the feature, =0, =0; mean(x) represents the mean value of the feature, std(x) represents the standard deviation of the feature, and a represents The weight parameter, b represents The weight parameter of .

30. The device according to claim 28, wherein The initial value of the right boundary RB in the at least one piecewise function in the second stage satisfies the following formula: ； The initial value of the left boundary LB of the at least one piecewise function in the second stage satisfies the following formula: ； in, represents the predicted average value of the features obtained after the last iteration of the first stage; represents the predicted standard deviation of the feature obtained after the last iteration of the first stage, and c represents the parameter.

31. The device according to claim 26, characterized in that The multiple trainable parameters in the at least one piecewise function include: a right boundary, a left boundary, a slope on an interval with a maximum domain, a slope on an interval with a minimum domain, or a function value corresponding to a dividing point.

32. The device according to claim 25 or 26, characterized in that The number of segments of the piecewise function is any value from 6 to 18.

33. A training device for a neural network model, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 8.

34. A data processing device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 9 to 16.

35. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, the program codes including codes for executing the method according to any one of claims 1 to 8 or 9 to 16.

36. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 8 or 9 to 16.

Images