Activation function selection method and device, equipment, storage medium and program product

Abstract

The invention relates to an activation function selection method and device, equipment, a storage medium and a program product. The method comprises the steps that an initial seed graph is constructed according to input data of each network layer of a target neural network in the last round, each piece of input data corresponds to an initial seed value in the initial seed graph, iteration processing is conducted on the initial seed values according to adjacent seed values of the initial seed values in the initial seed graph, a target seed graph is obtained, and the initial seed values of the target neural network are obtained; according to the mapping relation between the target seed graph and an activation function distribution graph associated with the target neural network, target selected parameters of all the candidate activation functions are determined, the activation function distribution graph comprises all types of candidate activation functions associated with the target neural network, and according to the target selected parameters of all the candidate activation functions, all the types of candidate activation functions associated with the target neural network are selected. And selecting a target activation function of the target neural network in the current round from the candidate activation functions. By adopting the method, the generalization ability of the neural network can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an activation function selection method, device, equipment, storage medium and program product. Background Art

[0002] With the continuous development of artificial intelligence, neural network technology has become a hot research topic in the field of artificial intelligence. The activation function in the neural network is used to transform the output data of each layer of the neural network, thereby injecting nonlinear factors into the entire network structure, so that the neural network can fit various curves, thereby enhancing the expression ability of the neural network.

[0003] However, although the existing activation functions of neural networks provide certain nonlinear factors for neural networks, a single activation function will reduce the generalization ability of the neural network model during the operation of the neural network. Summary of the invention

[0004] Based on this, it is necessary to provide an activation function selection method, device, equipment, storage medium and program product that can improve the generalization ability of neural networks in response to the above technical problems.

[0005] In a first aspect, the present application provides an activation function selection method, comprising:

[0006] Construct an initial seed graph based on the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0007] Iteratively process the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map;

[0008] Determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0009] According to the target selection parameters of each candidate activation function, the target activation function of the target neural network in this round is selected from each candidate activation function.

[0010] In an embodiment of the present application, by iteratively processing the input data of each network layer in the previous round as the initial seed, and selecting the target activation function of this round from an activation function distribution map containing multiple candidate activation functions according to the iteration result, the randomness of the activation function selection can be guaranteed, thereby improving the generalization ability of the neural network.

[0011] In one embodiment, constructing an initial seed graph according to input data of each network layer of the target neural network in the previous round includes:

[0012] According to the total amount of input data of each network layer of the target neural network in the previous round, an initial matrix is ​​constructed; according to the input data of each network layer in the previous round, the data threshold corresponding to each network layer is determined; according to the size relationship between the input data of each network layer in the previous round and the corresponding data threshold, the initial seed value corresponding to each input data is selected from the candidate seed values; the initial seed value corresponding to each input data is written into the initial matrix to obtain an initial seed graph.

[0013] In an embodiment of the present application, by selecting the initial seed value corresponding to each input data from the candidate seed values ​​according to the size relationship between the input data of each network layer and the corresponding data threshold, and then constructing the initial seed graph, the rationality of the construction of the initial seed graph can be guaranteed.

[0014] In one embodiment, according to the input data of each network layer of the target neural network in the previous round, the data threshold corresponding to each network layer is determined, including:

[0015] For each network layer in the target neural network, the mean of the input data corresponding to the network layer in the previous round is used as the data threshold corresponding to the network layer.

[0016] In the embodiment of the present application, by taking the average of the input data in each network layer as the data threshold corresponding to the network layer, the rationality of the determination of the data threshold can be ensured.

[0017] In one embodiment, the activation function distribution map is presented in a matrix form, and each matrix element corresponds to a candidate activation function;

[0018] According to the mapping relationship between the target seed map and the activation function distribution map associated with the target neural network, the target selected parameters of each candidate activation function are determined, including:

[0019] According to the mapping relationship between the target seed map and the activation function distribution map, the seed area corresponding to each matrix element in the activation function distribution map is determined; according to the seed area corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map, the target selection parameters of each candidate activation function are determined.

[0020] In an embodiment of the present application, by presenting the activation function distribution map in matrix form, and determining the seed area corresponding to each matrix element in the activation function distribution map based on the mapping relationship between the target seed map and the activation function distribution map, and then determining the target selection parameters of each candidate activation function, the rationality of the determination of the target selection parameters can be ensured.

[0021] In one embodiment, determining target selection parameters of each candidate activation function according to the seed region corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map includes:

[0022] For each matrix element, the initial selected parameters of the matrix element are determined according to the target seed value in the seed area corresponding to the matrix element; for each candidate activation function, the matrix element corresponding to the candidate activation function is determined according to the distribution information of the candidate activation function in the activation function distribution map, and the sum of the initial selected parameters of the matrix elements corresponding to the candidate activation function is used as the target selected parameters of the candidate activation function.

[0023] In the embodiment of the present application, by taking the sum of the initial selected parameters of each matrix element corresponding to the candidate activation function as the target selected parameter of the candidate activation function, the accuracy of the determination of the target selected parameter can be ensured.

[0024] In one embodiment, according to target selection parameters of each candidate activation function, selecting a target activation function of the target neural network in this round from each candidate activation function includes:

[0025] The candidate activation function with the largest target selected parameter among all candidate activation functions is used as the target activation function of the target neural network in this round.

[0026] In the embodiment of the present application, by selecting the candidate activation function with the largest target parameter as the target activation function of the target neural network in this round, the rationality of the determination of the target activation function can be ensured.

[0027] In a second aspect, the present application also provides an activation function selection device, comprising:

[0028] A construction module, used to construct an initial seed graph according to the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0029] An iteration module, used for iteratively processing the initial seed value according to adjacent seed values ​​of the initial seed value in the initial seed map to obtain a target seed map;

[0030] A parameter determination module, used to determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0031] The function selection module is used to select parameters according to the targets of each candidate activation function, and select the target activation function of the target neural network in this round from each candidate activation function.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Construct an initial seed graph based on the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0034] Iteratively process the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map;

[0035] Determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0036] According to the target selection parameters of each candidate activation function, the target activation function of the target neural network in this round is selected from each candidate activation function.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Construct an initial seed graph based on the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0039] Iteratively process the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map;

[0040] Determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0041] According to the target selection parameters of each candidate activation function, the target activation function of the target neural network in this round is selected from each candidate activation function.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Construct an initial seed graph based on the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0044] Iteratively process the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map;

[0045] Determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0046] According to the target selection parameters of each candidate activation function, the target activation function of the target neural network in this round is selected from each candidate activation function.

[0047] The above activation function selection method, device, equipment, storage medium and program product construct an initial seed map according to the input data of each network layer of the target neural network in the previous round, and iteratively process the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map; then, according to the mapping relationship between the target seed map and the activation function distribution map associated with the target neural network, determine the target selection parameters of each candidate activation function, and then select the target activation function of the target neural network in this round from each candidate activation function according to the target selection parameters of each candidate activation function. Compared with the related art, which directly selects a fixed activation function as the activation function of the neural network, the above method uses the input data of each network layer in the previous round as the initial seed for iterative processing, and selects the target activation function of this round from the activation function distribution map containing multiple candidate activation functions according to the iterative results, which can ensure the randomness of the activation function selection and thus improve the generalization ability of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of an activation function selection method in one embodiment;

[0050] Figure 2 A schematic diagram of a process of constructing an initial seed graph in one embodiment;

[0051] Figure 3 A schematic diagram of a process for determining target selection parameters in one embodiment;

[0052] Figure 4 A schematic diagram of a process for determining target selection parameters in another embodiment;

[0053] Figure 5 is a target seed map in one embodiment;

[0054] Figure 6 is an activation function distribution diagram in one embodiment;

[0055] Figure 7 is a mapping relationship diagram in an embodiment;

[0056] Figure 8 A schematic flow chart of an activation function selection method in another embodiment;

[0057] Fig. 9 is a structural block diagram of an activation function selection device in one embodiment;

[0058] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] With the continuous development of artificial intelligence, neural network technology has become a hot research topic in the field of artificial intelligence. The activation function in the neural network is used to transform the output data of each layer of the neural network, thereby injecting nonlinear factors into the entire network structure, so that the neural network can fit various curves, thereby enhancing the expression ability of the neural network.

[0061] However, although the existing activation functions of neural networks provide certain nonlinear factors for neural networks, a single activation function will reduce the generalization ability of the neural network model during the operation of the neural network.

[0062] Based on this, in an exemplary embodiment, Figure 1 As shown, an activation function selection method is provided, and the method is applied to a function selection device deployed in a neural network model as an example for explanation, and specifically includes the following steps:

[0063] S101, constructing an initial seed graph according to the input data of each network layer of the target neural network in the previous round.

[0064] Among them, the so-called target neural network is used to characterize the neural network that needs to select an activation function; the initial seed is the random seed in the initial state. Furthermore, the random seed is an object that uses a true random number as an initial condition and then uses a certain algorithm to continuously iterate to generate random numbers; each input data corresponds to an initial seed value in the initial seed map.

[0065] In order to ensure the flexibility of activation function selection, before the target neural network is used in this round, an activation function required for this round can be selected from the pre-configured candidate activation functions based on the input data of each network layer in the previous round.

[0066] In an optional implementation, the input data of each network layer of the target neural network in the previous round can be directly input into a trained seed graph construction model, and the seed graph construction model outputs an initial seed graph based on the input data and model parameters.

[0067] In another optional implementation, each input data can be used as an initial seed. For each input data, the input data can be input into a trained initial seed value determination model. The initial seed value determination model can determine the initial seed value corresponding to the input data based on the input data and model parameters.

[0068] After determining the initial seed value corresponding to each input data, the positional relationship between each input data can be determined according to the position of the network node corresponding to the input data of each network layer in the target neural network; then, the initial seed graph is constructed according to the positional relationship between each input data and the initial seed value of the initial seed corresponding to each input data.

[0069] For example, an empty matrix diagram may be constructed according to the positional relationship between each input data; and then the initial seed value corresponding to each input data is filled into the empty matrix diagram to obtain the initial seed diagram.

[0070] S102, iteratively processing the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain a target seed map.

[0071] The seed values ​​at adjacent positions of each initial seed value in the initial seed map are the adjacent seed values ​​of the initial seed value; the so-called target seed map is the seed map obtained after iterative processing of the initial seed values.

[0072] In an optional implementation, an iterative model may be trained based on a preset iterative rule, the initial seed map may be directly input into the iterative model, and the iterative model may output a target seed map according to the initial seed map and model parameters.

[0073] In another optional implementation, for each initial seed value in the initial seed map, the adjacent seed values ​​of the initial seed value can be determined according to the position of the initial seed value in the initial seed map; then, based on a preset iteration rule, the change mode of the initial seed value in the next iteration round is determined according to the adjacent seed values ​​of the initial seed value. After multiple iterations, the target seed value corresponding to the initial seed value can be determined.

[0074] Furthermore, after the target seed values ​​corresponding to the initial seed values ​​are determined, the initial seed values ​​in the initial seed map are updated to obtain the target seed map.

[0075] S103, determining target selection parameters of each candidate activation function according to a mapping relationship between the target seed map and the activation function distribution map associated with the target neural network.

[0076] The so-called candidate activation function is the activation function that the target neural network can select; the so-called target selection parameter is used to characterize the selection probability of each candidate activation function. Furthermore, the larger the target selection parameter, the higher the probability of function selection.

[0077] The activation function distribution graph includes candidate activation functions of various types associated with the target neural network, and the distribution number of each candidate activation function in the activation function distribution graph is the same.

[0078] Before the target neural network is put into use, in order to ensure the rationality of the activation function selection, the same number of candidate activation functions can be randomly filled into the initial function distribution map that has a mapping relationship with the target seed map to obtain the activation function distribution map.

[0079] For example, the S-type activation function sigmoid, the linear rectification function relu, the variant gated linear activation function swiglu and the Gaussian error linear unit activation function gelu can be used as candidate activation functions at the same time; then, the activation function distribution map can be constructed by using the same number of sigmoid, relu, swiglu and gelu.

[0080] In an optional implementation, the target seed corresponding to each candidate activation function in the activation function distribution map can be determined from the target seed map based on the mapping relationship between the target seed map and the activation function distribution map; then, the target selection parameter corresponding to the candidate activation function is determined based on the target seed values ​​of all target seeds corresponding to the same candidate activation function.

[0081] For example, the sum of all target seed values ​​corresponding to the same candidate activation function may be used as the target selection parameter corresponding to the candidate activation function.

[0082] In another optional implementation, a parameter determination model may be trained based on the distribution of candidate activation functions in the activation function distribution map and the mapping relationship between the target seed map and the activation function distribution map.

[0083] Furthermore, the target seed map can be directly input into the parameter determination model, and the parameter determination model outputs the target selected parameters of each candidate activation function according to each target seed value and the model parameters in the target seed map.

[0084] S104, selecting a target activation function of the target neural network in this round from among the candidate activation functions according to the target selection parameters of each candidate activation function.

[0085] It can be understood that since the target selection parameter can reflect the selection probability of the candidate activation function to a certain extent, in one embodiment, the candidate activation function with the largest target selection parameter among the candidate activation functions can be used as the target activation function of the target neural network in this round.

[0086] Exemplarily, the candidate activation functions may be sorted in order from large to small based on the target selection parameters, and the candidate activation function ranked first may be used as the target activation function of the target neural network in this round.

[0087] In order to ensure the diversity of activation function selection, in another possible implementation mode, the target activation function corresponding to the previous round can be first eliminated from each candidate activation function, and then the candidate activation function with the largest target selection parameter is selected from the remaining candidate activation functions as the target activation function of the target neural network in this round.

[0088] In the above activation function selection method, an initial seed map is constructed according to the input data of each network layer of the target neural network in the previous round, and the initial seed value is iteratively processed according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain the target seed map; then, the target selection parameters of each candidate activation function are determined according to the mapping relationship between the target seed map and the activation function distribution map associated with the target neural network, and then the target activation function of the target neural network in this round is selected from each candidate activation function according to the target selection parameters of each candidate activation function. Compared with the related art, in which a fixed activation function is directly selected as the activation function of the neural network, the above method is adopted to iteratively process the input data of each network layer in the previous round as the initial seed, and select the target activation function of this round from the activation function distribution map containing multiple candidate activation functions according to the iteration result, which can ensure the randomness of the activation function selection and thus improve the generalization ability of the neural network.

[0089] In order to ensure the rationality of the construction of the initial seed graph, based on the above embodiment, in the embodiment of the present application, an optional method for constructing the initial seed graph is provided, such as Figure 2 As shown, the specific steps include:

[0090] S201, constructing an initial matrix according to the total amount of input data of each network layer of the target neural network in the previous round.

[0091] The so-called initial matrix is ​​an idle matrix without any numerical values ​​filled in.

[0092] In one possible implementation, the row and column parameters of the matrix can be determined based on the total amount of input data of each network layer of the target neural network in the previous round, thereby constructing an initial matrix.

[0093] For example, in order to facilitate subsequent data mapping, a square matrix can be used as the initial matrix; then, according to the total amount of input data of each network layer, the row and column parameters whose product of the number of rows and columns is greater than or equal to the total amount of data are determined. That is, the total amount of data is squared and rounded up to obtain the row and column parameters of the matrix.

[0094] For example, when the total amount of input data of each network layer is between 65 and 81, it can be determined that the row and column parameters of the initial matrix are 9, that is, the initial matrix is ​​a 9*9 matrix.

[0095] S202, determining a data threshold corresponding to each network layer according to input data of each network layer in the previous round.

[0096] The so-called data threshold is a value that can measure the size of input data.

[0097] It is understandable that due to the large numerical differences between the input data, directly filling the input data into the initial matrix will increase the difficulty of determining the selected parameters. Therefore, for each network layer, the data threshold corresponding to the network layer can be determined in advance based on the input data of the network layer in the previous round.

[0098] In one possible implementation, the input data of the network layer in the previous round can be directly input into a trained threshold determination model, and the threshold determination model outputs the data threshold corresponding to the network layer based on the input data and model parameters of the network layer.

[0099] In another possible implementation, for each network layer in the target neural network, the mean of each input data corresponding to the network layer in the previous round is used as the data threshold corresponding to the network layer.

[0100] For example, for each network layer in the target neural network, the total input value of the network layer can be determined based on the sum of the input data corresponding to the network layer in the previous round; then, the total input value corresponding to the network layer is divided by the number of input data to obtain the data threshold corresponding to the network layer.

[0101] S203, selecting an initial seed value corresponding to each input data from candidate seed values ​​according to the size relationship between the input data of each network layer in the previous round and the corresponding data threshold.

[0102] The so-called candidate seed value is a value that can represent the selected state of the candidate activation function. For example, the candidate seed value may include "0" and "1", and "0" is used to represent that the candidate activation function is not selected; "1" is used to represent that the candidate activation function is selected.

[0103] In an optional implementation, for each input data in the previous round, an initial seed value corresponding to the input data can be selected from candidate seed values ​​based on the size relationship between the input data and a data threshold of the network layer where the input data is located.

[0104] For example, when the candidate seed values ​​include "0" and "1", if the input data is greater than or equal to the data threshold of the network layer where the input data is located, the initial seed value corresponding to the input data is "1"; if the input data is less than the data threshold of the network layer where the input data is located, the initial seed value corresponding to the input data is "0".

[0105] S204, writing the initial seed value corresponding to each input data into the initial matrix to obtain an initial seed map.

[0106] In an optional implementation, after determining the initial seed value of each input data, the initial seed value corresponding to each input data can be written into the initial matrix in sequence according to the relationship between the input positions of each input data in the target neural network to obtain an initial seed map.

[0107] It can be understood that when the number of matrix cells in the initial matrix is ​​greater than the number of input data, the initial seed values ​​corresponding to the input data can be written into the initial matrix in a scattered manner, and then the empty matrix cells are randomly filled with candidate seed values ​​to obtain an initial seed map.

[0108] For example, when the total amount of input data is 75, after the initial seed values ​​corresponding to the input data are scattered and written into the 9*9 initial matrix, there will be 6 empty matrix cells. At this time, 3 "0"s and 3 "1"s can be used to randomly fill the empty matrix cells to obtain the initial seed map.

[0109] In an embodiment of the present application, by selecting the initial seed value corresponding to each input data from the candidate seed values ​​according to the size relationship between the input data of each network layer and the corresponding data threshold, and then constructing the initial seed graph, the rationality of the construction of the initial seed graph can be guaranteed.

[0110] In order to ensure the rationality of the target selection parameters, on the basis of the above embodiment, in order to facilitate the construction of a mapping relationship between the activation function distribution map and the target seed map, in the embodiment of the present application, the activation function distribution map is presented in the form of a matrix, and each matrix element corresponds to a candidate activation function; based on this, an optional method for determining the target selection parameters is provided, such as Figure 3 As shown, the specific steps include:

[0111] S301, determining a seed region corresponding to each matrix element in the activation function distribution map according to a mapping relationship between the target seed map and the activation function distribution map.

[0112] Since both the activation function distribution map and the target seed map are presented in matrix form, in an optional implementation, the mapping relationship between the target seed map and the activation function distribution map can be determined based on the row and column parameters of the target seed map and the row and column parameters of the activation function distribution map.

[0113] Furthermore, for each matrix element in the activation function distribution map, the seed region corresponding to the matrix element can be determined based on the mapping relationship between the target seed map and the activation function distribution map and according to the position information of the matrix element in the activation function distribution map.

[0114] For example, when the target seed map is a 9*9 matrix and the activation function distribution map is a 3*3 matrix, it can be determined that each matrix element in the activation function distribution map corresponds to a 3*3 seed region in the target seed map, wherein the seed regions corresponding to each matrix element do not overlap.

[0115] At this time, for the matrix element A located in the first row and first column (1,1) in the activation function distribution diagram, the seed area A' corresponding to the matrix element A includes {(1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3)}, that is, the first 3*3 seed area in the upper left corner.

[0116] S302, determining target selection parameters of each candidate activation function according to the seed region corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map.

[0117] The so-called distribution information is used to characterize the position information of each candidate activation function in the activation function distribution map.

[0118] In an optional implementation, the matrix element corresponding to each candidate activation function may be determined based on the distribution information of each candidate activation function in the activation function distribution map, and then the seed region associated with the matrix element corresponding to each candidate activation function may be determined.

[0119] Furthermore, for each candidate activation function, a target selection parameter of the candidate activation function may be determined according to the target seed values ​​in all seed regions corresponding to the candidate activation function. For example, the sum of the target seed values ​​in all seed regions corresponding to the candidate activation function may be used as the target selection parameter of the candidate activation function.

[0120] In an embodiment of the present application, by presenting the activation function distribution map in matrix form, and determining the seed area corresponding to each matrix element in the activation function distribution map based on the mapping relationship between the target seed map and the activation function distribution map, and then determining the target selection parameters of each candidate activation function, the rationality of the determination of the target selection parameters can be ensured.

[0121] In order to further ensure the accuracy of the target selection parameters, based on the above embodiments, another optional method for determining the target selection parameters is provided in the embodiments of the present application, such as Figure 4 As shown, the specific steps include:

[0122] S401, for each matrix element, determining an initial selection parameter of the matrix element according to a target seed value in a seed region corresponding to the matrix element.

[0123] Among them, the so-called initial selected parameters are used to characterize the selection probability of the candidate activation function corresponding to the matrix elements.

[0124] In an optional implementation, for each matrix element, the sum of the target seed values ​​in the seed region corresponding to the matrix element may be used as the initial selection parameter of the matrix element.

[0125] In another optional implementation, for each matrix element, the number of target seed values ​​that can represent the selected state in the seed region corresponding to the matrix element can be used as the initial selection parameter of the matrix element.

[0126] For example, when the number of target seed values ​​"1" in the seed area B' associated with the matrix element B is 3, the initial selection parameter of the matrix element B is 3; similarly, all the target seed values ​​in the seed area B' can be added together. Since the other target seed values ​​except "1" are "0", the sum of the target seed values ​​in the seed area B' is also 3.

[0127] S402, for each candidate activation function, determine the matrix element corresponding to the candidate activation function according to the distribution information of the candidate activation function in the activation function distribution map, and use the sum of the initial selected parameters of each matrix element corresponding to the candidate activation function as the target selected parameter of the candidate activation function.

[0128] In an optional implementation, after determining the matrix elements corresponding to each candidate activation function based on the distribution information of each candidate activation function in the activation function distribution map, for each candidate activation function, the sum of the initial selected parameters of the matrix elements corresponding to the candidate activation function can be used as the target selected parameters of the candidate activation function.

[0129] In the embodiment of the present application, by taking the sum of the initial selected parameters of each matrix element corresponding to the candidate activation function as the target selected parameter of the candidate activation function, the accuracy of the determination of the target selected parameter can be ensured.

[0130] In order to ensure the logic of the activation function selection, based on the above embodiment, in the embodiment of the present application, based on Conway's Game of Life, for the input data of 81 dimensions in the previous round, an optional method for determining the target activation function is provided, which specifically includes the following steps:

[0131] After obtaining the input data of 81 dimensions in the previous round, you can refer to the above step S202 to determine the data threshold corresponding to each network layer according to the input data of each network layer; then, for each input data, if the input data is greater than or equal to the data threshold of the network layer corresponding to the input data layer, the initial seed value of the input data is "1"; if the input data is less than the data threshold of the network layer corresponding to the input data layer, the initial seed value of the input data is "0".

[0132] After the initial seed value of each input data is determined, the initial seed value of each input data can be filled into a 9*9 initial matrix to obtain an initial seed map.

[0133] In the present application, the iteration rule of Conway's Game of Life is: when an initial seed value is "0", if there are 3 "1"s in the adjacent seed values, the initial seed value will become "1" in the next round of iteration.

[0134] When an initial seed value is "1", if there are more than 3 "1"s in the adjacent seed values, the seed value of this initial seed value will become "0" in the next round of iteration; if there are 2-3 "1"s in the adjacent seed values, the seed value of this initial seed value will remain unchanged in the next round of iteration, that is, "1"; if there are less than 2 "1"s in the adjacent seed values, the seed value of this initial seed value will become "0" in the next round of iteration.

[0135] Based on Conway's Game of Life, after iterating the initial seed values ​​in the initial seed graph for a preset number of times, we can get the following: Figure 5 The target seed map shown.

[0136] In this application, there are three candidate activation functions in the target neural network: sigmoid, relu and swiglu, which can be constructed as follows Figure 6 The activation function distribution diagram shown in Figure 1 shows that there are the same number of sigmoid, relu, and swiglu in the activation function distribution diagram.

[0137] In order to ensure the rationality of the mapping relationship between the target seed map and the activation function distribution map, the following can be generated: Figure 7 The mapping relationship diagram is shown in the figure. Among them, "1" is used to indicate that the candidate activation function has been selected; "0" is used to indicate that the candidate activation function has not been selected.

[0138] Based on this, it can be determined that the initial selected parameter of the "sigmoid" element in the first row and first column is 4; the initial selected parameter of the "relu" element in the first row and second column is 3; the initial selected parameter of the "swiglu" element in the first row and third column is 4; the initial selected parameter of the "relu" element in the second row and first column is 3; the initial selected parameter of the "swiglu" element in the second row and second column is 5; the initial selected parameter of the "sigmoid" element in the second row and third column is 3; the initial selected parameter of the "swiglu" element in the third row and first column is 4; the initial selected parameter of the "sigmoid" element in the second row and third column is 4; the initial selected parameter of the "relu" element in the third row and third column is 4.

[0139] In summary, the target parameter of "sigmoid" is 11; the target parameter of "relu" element is 10; the target parameter of "swiglu" element is 13. Therefore, the candidate activation function swiglu can be selected as the target activation function of this round.

[0140] In the embodiment of the present application, the target activation function of this round is selected by iterating the initial seed graph based on the Conway Game of Life, so that the logic of the activation function selection can be guaranteed.

[0141] Figure 8 FIG. 2 is a flow chart of an activation function selection method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of an activation function selection method. Figure 8 The specific implementation process is as follows:

[0142] S801, constructing an initial matrix according to the total amount of input data of each network layer of the target neural network in the previous round.

[0143] S802, for each network layer in the target neural network, the mean of each input data corresponding to the network layer in the previous round is used as the data threshold corresponding to the network layer.

[0144] S803, selecting an initial seed value corresponding to each input data from candidate seed values ​​according to the size relationship between the input data of each network layer in the previous round and the corresponding data threshold.

[0145] S804, writing the initial seed value corresponding to each input data into the initial matrix to obtain an initial seed graph.

[0146] S805, performing iterative processing on each initial seed value according to adjacent seed values ​​of each initial seed value in the initial seed map to obtain a target seed map.

[0147] S806: Present an activation function distribution diagram in a matrix form according to each candidate activation function.

[0148] S807, determining the seed region corresponding to each matrix element in the activation function distribution map according to the mapping relationship between the target seed map and the activation function distribution map.

[0149] S808, for each matrix element, determine the initial selection parameter of the matrix element according to the target seed value in the seed region corresponding to the matrix element.

[0150] S809: For each candidate activation function, determine the matrix element corresponding to the candidate activation function according to the distribution information of the candidate activation function in the activation function distribution map.

[0151] S810, taking the sum of the initial selected parameters of each matrix element corresponding to each candidate activation function as the target selected parameter of the candidate activation function.

[0152] S811, taking the candidate activation function with the largest target selected parameter among the candidate activation functions as the target activation function of the target neural network in this round.

[0153] The specific process of the above S801-S811 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present application also provides an activation function selection device for implementing the activation function selection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more activation function selection device embodiments provided below can refer to the limitations of the activation function selection method above, and will not be repeated here.

[0156] In an exemplary embodiment, Fig. 9 As shown, an activation function selection device 1 is provided, comprising: a construction module 10, an iteration module 20, a parameter determination module 30 and a function selection module 40, wherein:

[0157] A construction module 10 is used to construct an initial seed graph according to the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph;

[0158] Iteration module 20, used for iteratively processing the initial seed value according to the adjacent seed values ​​of the initial seed value in the initial seed map to obtain a target seed map;

[0159] A parameter determination module 30 is used to determine target selected parameters of each candidate activation function according to a mapping relationship between a target seed map and an activation function distribution map associated with a target neural network; wherein the activation function distribution map includes candidate activation functions of various types associated with the target neural network;

[0160] The function selection module 40 is used to select the target activation function of the target neural network in this round from the candidate activation functions according to the target selection parameters of each candidate activation function.

[0161] In an exemplary embodiment, the building block 10 includes:

[0162] A matrix construction unit, used to construct an initial matrix according to the total amount of input data of each network layer of the target neural network in the previous round;

[0163] A threshold determination unit, used to determine the data threshold corresponding to each network layer according to the input data of each network layer in the previous round;

[0164] A selection unit, used to select an initial seed value corresponding to each input data from candidate seed values ​​according to the size relationship between the input data of each network layer in the previous round and the corresponding data threshold;

[0165] The writing unit is used to write the initial seed value corresponding to each input data into the initial matrix to obtain an initial seed map.

[0166] In an exemplary embodiment, the threshold determination unit is specifically configured to:

[0167] For each network layer in the target neural network, the mean of the input data corresponding to the network layer in the previous round is used as the data threshold corresponding to the network layer.

[0168] In an exemplary embodiment, the activation function distribution diagram is presented in a matrix form, and each matrix element corresponds to a candidate activation function; the parameter determination module 30 includes:

[0169] A region determination unit, used to determine the seed region corresponding to each matrix element in the activation function distribution map according to the mapping relationship between the target seed map and the activation function distribution map;

[0170] The parameter determination unit is used to determine the target selection parameters of each candidate activation function according to the seed area corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map.

[0171] In an exemplary embodiment, the parameter determination unit is specifically configured to:

[0172] For each matrix element, the initial selected parameters of the matrix element are determined according to the target seed value in the seed area corresponding to the matrix element; for each candidate activation function, the matrix element corresponding to the candidate activation function is determined according to the distribution information of the candidate activation function in the activation function distribution map, and the sum of the initial selected parameters of the matrix elements corresponding to the candidate activation function is used as the target selected parameters of the candidate activation function.

[0173] In an exemplary embodiment, the function selection module 40 is specifically used to:

[0174] The candidate activation function with the largest target selected parameter among all candidate activation functions is used as the target activation function of the target neural network in this round.

[0175] Each module in the activation function selection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0176] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store input data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an activation function selection method is implemented.

[0177] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0178] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0179] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0180] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0181] It should be noted that the data involved in this application (including but not limited to input data of each network layer, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0182] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for selecting an activation function, characterized in that: The method comprises: Constructing an initial seed graph according to the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph; Iteratively process the initial seed value according to adjacent seed values ​​of the initial seed value in the initial seed map to obtain a target seed map; Determine target selection parameters of each candidate activation function according to a mapping relationship between the target seed graph and an activation function distribution graph associated with the target neural network; wherein the activation function distribution graph includes candidate activation functions of various types associated with the target neural network; According to the target selection parameters of each candidate activation function, the target activation function of the target neural network in this round is selected from each candidate activation function.

2. The method according to claim 1, characterized in that: The initial seed graph is constructed according to the input data of each network layer of the target neural network in the previous round, including: Construct an initial matrix based on the total amount of input data of each network layer of the target neural network in the previous round; Determine the data threshold corresponding to each network layer according to the input data of each network layer in the previous round; According to the size relationship between the input data of each network layer in the previous round and the corresponding data threshold, select the initial seed value corresponding to each input data from the candidate seed values; The initial seed value corresponding to each input data is written into the initial matrix to obtain an initial seed map.

3. The method according to claim 2, characterized in that Determining the data threshold corresponding to each network layer according to the input data of each network layer of the target neural network in the previous round includes: For each network layer in the target neural network, the mean of the input data corresponding to the network layer in the previous round is used as the data threshold corresponding to the network layer.

4. The method according to claim 2, characterized in that: The activation function distribution diagram is presented in matrix form, and each matrix element corresponds to a candidate activation function; The step of determining target selected parameters of each candidate activation function according to a mapping relationship between the target seed graph and the activation function distribution graph associated with the target neural network includes: Determine, according to a mapping relationship between the target seed map and the activation function distribution map, a seed region corresponding to each matrix element in the activation function distribution map; According to the seed area corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map, the target selection parameter of each candidate activation function is determined.

5. The method according to claim 4, characterized in that Determining target selection parameters of each candidate activation function according to the seed area corresponding to each matrix element in the activation function distribution map and the distribution information of each candidate activation function in the activation function distribution map includes: For each matrix element, determining an initial selection parameter of the matrix element according to a target seed value in a seed region corresponding to the matrix element; For each candidate activation function, the matrix element corresponding to the candidate activation function is determined according to the distribution information of the candidate activation function in the activation function distribution map, and the sum of the initial selected parameters of the matrix elements corresponding to the candidate activation function is used as the target selected parameter of the candidate activation function.

6. The method according to claim 2, characterized in that The step of selecting a target activation function of the target neural network in this round from among the candidate activation functions according to the target parameters of the candidate activation functions includes: The candidate activation function with the largest target selected parameter among the candidate activation functions is used as the target activation function of the target neural network in this round.

7. An activation function selection device, characterized in that: The device comprises: A construction module, used to construct an initial seed graph according to the input data of each network layer of the target neural network in the previous round; wherein each input data corresponds to an initial seed value in the initial seed graph; An iteration module, configured to iteratively process the initial seed value according to adjacent seed values ​​of the initial seed value in the initial seed map to obtain a target seed map; A parameter determination module, used to determine the target selected parameters of each candidate activation function according to the mapping relationship between the target seed map and the activation function distribution map associated with the target neural network; wherein the activation function distribution map contains candidate activation functions of various types associated with the target neural network; The function selection module is used to select parameters according to the targets of each candidate activation function, and select the target activation function of the target neural network in this round from each candidate activation function.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images