Neural network training method and device, text translation method, electronic device and medium
By using activation functions max(0, a(xc), b(xd)) and pruning operations in neural networks, the problem of high hardware resource consumption is solved, achieving efficient neural network training and text translation, and improving computational accuracy and sparsity.
Patent Information
- Application Number
- CN202210260924.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-16
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-03-16
AI Technical Summary
In existing technologies, when neural network algorithms are implemented on edge devices such as FPGAs or ASICs, the computing power is insufficient and the hardware resources are consumed in large quantities. Traditional activation functions such as ReLU functions have limited representation capabilities and consume a lot of resources, which cannot meet the requirements for computing accuracy and latency.
The activation function max(0, a(xc), b(xd)) is adopted, where a and b are the slopes and c and d are the zeros. It is implemented in hardware through addition and shift operations, and combined with pruning operations to improve network sparsity and computational efficiency.
It improves the computational accuracy and sparsity of neural networks with low resource consumption, reduces storage space and data transmission requirements, and enhances algorithm performance.
Smart Images
Figure CN116822608B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network training method and device, a text translation method, an electronic device, and a medium. BACKGROUND
[0002] At present, artificial intelligence algorithms such as the Transformer network in the NLP field and the calculation in the edge device field such as FPGA or ASIC need to meet the characteristics of high calculation accuracy and small calculation delay under the condition of low resource and power consumption, and the calculation ability of the related technology cannot meet the actual application such as text translation. The main reason restricting the calculation ability is the complexity of the network, and some operations are not suitable for hardware design or some functions have high complexity when using hardware design, and the corresponding hardware design cost is high, which cannot meet the FPGA or ASIC deployment requirements of the algorithm, and thus it is necessary to optimize the algorithm for each structure of the hardware implementation.
[0003] At present, in order to introduce a nonlinear operation, most neural network algorithms in the technical field of text translation need to use a large number of activation functions, and for traditional activation functions such as the Relu function suitable for hardware implementation, although the implementation cost is low, the representation ability is limited, the flexibility is insufficient, and the hardware resource consumption is large. SUMMARY
[0004] The main purpose of the present application is to provide a neural network training method and device, a text translation method, an electronic device, and a medium, which aims to solve the technical problem of large hardware resource consumption in the prior art.
[0005] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present application provides a neural network training method, comprising:
[0006] obtaining an initial training sample data set, wherein the initial training sample in the initial training sample data set comprises initial text data;
[0007] preprocessing the initial training sample to generate a training sample data set, wherein the training sample in the training sample data set comprises text data and label information corresponding to the text data, and the label information is used to represent the translation result of the text data;
[0008] constructing a neural network to be trained, wherein the neural network to be trained comprises an activation layer, a fully connected forward propagation layer, and a convolution layer, the activation layer is arranged between the fully connected forward propagation layers or after the convolution layer, and the activation function of the activation layer is implemented by using addition and shift operations;
[0009] training the neural network to be trained by using the training sample data set to obtain a trained neural network.
[0010] In an embodiment of the present disclosure, the activation function of the activation layer is max(0, a(x-c), b(x-d)), where x represents an activation value, a < b, the range of a and b is (0, 1), a and b are multiples of 0.125, c < d, and c and d are function zero points.
[0011] In an embodiment of the present disclosure, the method further comprises:
[0012] Obtaining a range of values of parameters a, b, c and d in the activation function max(0, a(x-c), b(x-d));
[0013] According to the range of values of a, b, c and d, setting a plurality of values of the activation function max(0, a(x-c), b(x-d));
[0014] Using the values of the plurality of activation functions max(0, a(x-c), b(x-d)), training the trained neural network to obtain the optimal value of the activation function max(0, a(x-c), b(x-d)).
[0015] In an embodiment of the present disclosure, the neural network includes any one of a CNN network model, a Transformer network model, an RNN network model, and an LSTM network model.
[0016] In an embodiment of the present disclosure, after obtaining the optimal value of the activation function max(0, a(x-c), b(x-d)), a pruning operation is performed.
[0017] A second aspect of the embodiments of the present application provides a text translation method, comprising:
[0018] Obtaining a sample data set to be translated, the sample data set to be translated including text data to be translated;
[0019] Inputting the text data to be translated into a neural network to obtain a text data translation result, the neural network being trained by the neural network training method provided in the first aspect of the embodiments of the present application.
[0020] A third aspect of the embodiments of the present application provides a neural network training device, comprising:
[0021] An obtaining module is configured to obtain an initial training sample data set, and initial training samples in the initial training sample data set include initial text data.
[0022] The preprocessing module is used to preprocess the initial training samples to generate a training sample dataset. The training samples in the training sample dataset include text data and label information corresponding to the text data. The label information is used to characterize the translation result of the text data.
[0023] A construction module is used to construct a neural network to be trained. The neural network to be trained includes an activation layer, a fully connected forward propagation layer, and a convolutional layer. The activation layer is set between the fully connected forward propagation layers or after the convolutional layer. The activation function of the activation layer is implemented using addition and shift operations.
[0024] The training module is used to train the neural network to be trained using the training sample dataset to obtain the trained neural network.
[0025] In one embodiment of this disclosure, the activation function of the activation layer is max(0, a(xc), b(xd)), where x represents the activation value, a < b, the range of a and b is (0, 1), and a and b are both multiples of 0.125, c < d, and c and d are both zeros of the function.
[0026] A third aspect of this application provides an electronic device, including:
[0027] A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements the neural network training method provided in the first aspect of the embodiments of this application or the text translation method provided in the second aspect of the embodiments of this application.
[0028] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the neural network training method provided in the first aspect of this application or the text translation method provided in the second aspect of this application. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 A schematic flowchart of a neural network training method provided in an embodiment of this application;
[0031] Figure 2 A schematic diagram showing the location of the activation layer in a neural network according to an embodiment of this application;
[0032] Figure 3 A schematic diagram of the position of the activation layer in the neural network according to an embodiment of the present application is shown.
[0033] Figure 4 A schematic diagram of the position of the activation layer in the neural network according to an embodiment of the present application is shown.
[0034] Figure 5 A flowchart of a text translation method according to an embodiment of the present application is shown.
[0035] Figure 6 A schematic diagram of the sparsity and loss function value experimental results according to an embodiment of the present application is shown.
[0036] Figure 7 A schematic diagram of the result performance of a text translation task according to an embodiment of the present application is shown.
[0037] Figure 8 A schematic diagram of the experimental results of the combination of the activation function and the pruning algorithm according to an embodiment of the present application is shown.
[0038] Figure 9 A structural schematic diagram of a neural network training device according to an embodiment of the present application is shown.
[0039] Figure 10 A hardware structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION
[0040] In order to make the application purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0041] Please refer to Figure 1 , Figure 1 A flowchart of a neural network training method according to an embodiment of the present application is shown. The method can be applied in an electronic device, and mainly includes the following steps:
[0042] S101, an initial training sample data set is obtained, and the initial training sample in the initial training sample data set includes initial text data;
[0043] S102, the initial training sample is preprocessed to generate a training sample data set, and the training sample in the training sample data set includes text data and label information corresponding to the text data. The label information is used to represent the translation result of the text data.
[0044] S103. Construct the neural network to be trained. The neural network to be trained includes activation layers, fully connected forward propagation layers, and convolutional layers. Activation layers are set between fully connected forward propagation layers or after convolutional layers. The activation function of the activation layer is implemented using addition and shift operations.
[0045] S104. Use the training sample dataset to train the neural network to be trained, and obtain the trained neural network.
[0046] In this disclosure, the input data for the activation layer (sparse_relu activation) is the output value of a neural network layer such as a CNN or a fully connected layer before the activation layer, or the value after sparse matrix compression, such as... Figure 2 As shown, this activation layer is located directly after the convolutional layer, as... Figure 3 As shown, this activation layer is located after the convolutional layer and the pooling layer, as... Figure 4 As shown, the activation layer is located between fully connected forward propagation layers (fully connected layers), and the activation function can be implemented in hardware using only shift and addition operations with extremely low resource consumption.
[0047] In one embodiment of this disclosure, the activation function of the activation layer is max(0, a(xc), b(xd)), where x represents the activation value, a < b, both a and b are in the range (0, 1), and both a and b are multiples of 0.125, c < d, and both c and d are zeros of the function. In this disclosure, the activation function with parameters a, b, c, and d is applied after the convolutional computation layer between fully connected forward propagation layers in a neural network, replacing the traditional activation function to provide nonlinear characteristics to the network. Furthermore, the activation function is implemented in hardware using shift and addition operations with extremely low resource consumption. The slope and zeros are both externally input, completing the neural network-based text translation task.
[0048] According to embodiments of this disclosure, the activation function can improve algorithm accuracy and sparsity of the neural network model from a software perspective, while reducing model storage space and data transmission volume. At the hardware level, the activation function can be implemented with extremely low resource consumption using shift and addition operations. In summary, the activation function of this disclosure, when applied to neural network algorithms, can improve algorithm accuracy and parameter sparsity, and can also be implemented with extremely low hardware consumption when hardware implementation is required.
[0049] In this disclosure, the activation function max(0, a(xc), b(xd)) is obtained by maximizing three sub-functions: y0 = 0, y1 = a(xc), and y2 = b(xd). Here, a and c are the parameters of sub-function y1, b and d are the parameters of sub-function y2, a and b are the slopes, and c and d are the zeros of the function. The slopes of each sub-function are multiples of 0.125, which can be obtained by shifting without multiplication in hardware. The slopes and zeros of each part can be adjusted during algorithm training, thus obtaining suitable parameters based on the algorithm training. The values of the above parameters a, b, c, and d are generally set to satisfy a < b and c < d, with a ranging from (0, 1) and b ranging from (0, 1).
[0050] According to embodiments of this disclosure, the sub-function y0 = 0 is essential, as it can improve the sparsity of network weights and activations. When determining parameters in the training algorithm, functions with larger zero points are preferred for the same effect. Ultimately, y0, y1, and y2 together constitute the activation function applied in the field of artificial intelligence technology.
[0051] In one embodiment of this disclosure, the method further includes: obtaining the value ranges of parameters a, b, c, and d in the activation function max(0, a(xc), b(xd)); setting multiple values for the activation function max(0, a(xc), b(xd)) based on the value ranges of a, b, c, and d; and training the trained neural network using the multiple values of the activation function max(0, a(xc), b(xd)) to obtain the optimal value of the activation function max(0, a(xc), b(xd)). The training process sets multiple parameter combinations, performs iterative experiments under the same conditions, and finds the hyperparameter that performs best in the combination, making the network model perform better than traditional activation functions. In one example, a = 0.25, b = 1, c = 0.2, d = 0.4, and the activation function is max(0, 0.25(x-0.2), x-0.4). In another example, a = 1, b = 1, c = 0.2, d = 0.2, and the activation function is max(0, x-0.2, x-0.2). In yet another example, a = 1, b = 1, c = 0, d = 0, and the activation function is max(0, x, x). In this case, the conventional activation function is a special case of the activation function shown in this disclosure.
[0052] In one embodiment of this disclosure, the activation function is applied to various neural networks in the field of artificial intelligence technology, including any one of CNN network models, Transformer network models, RNN network models, and LSTM network models.
[0053] In an embodiment of the present disclosure, after obtaining the optimal value of the activation function max(0, a(x-c), b(x-d)), the pruning operation is performed. In the present disclosure, after determining the optimal value in the activation function, the pruning operation is combined to improve the sparsity of the network model, which can improve the sparsity of the activation and weight, and facilitate subsequent hardware acceleration using sparse matrix.
[0054] Please refer to Figure 5 , Figure 5 The flowchart of the text translation method provided by an embodiment of the present application is shown in the figure. The method can be applied in an electronic device, and mainly includes the following steps:
[0055] S501, obtaining a sample data set to be translated, the sample data set to be translated including text data to be translated;
[0056] S502, inputting the text data to be translated into a neural network to obtain a text data translation result, the neural network being trained by the neural network training method shown in Figure 1 .
[0057] In the present disclosure, a=0.25, b=1, c=0.2, d=0.4, and the activation function is max(0, 0.25(x-0.2), x-0.4) are used for experiments. In the Transformer model, compared with the traditional activation function, the activation function of the present disclosure improves the forward layer sparsity by 150.95% in the case of higher accuracy. When using the iwslt14 data set with a total of more than 160,000 translation tasks for German-English translation tasks, compared with the traditional activation function, the translation result can be evaluated by using the Bleu score, and the Transformer model using the activation function of the present disclosure can obtain a score of 35.16, which is 0.87 higher than the Transformer model using the traditional activation function, and is improved by 2.54%. When combined with the pruning algorithm, the algorithm is randomly pruned in an unstructured manner to improve the sparsity of the parameters. After the pruning operation is combined with the sparse activation function of the present disclosure, the pruning can improve the weight sparsity to 78.26%, and in addition, the activation sparsity of the activation layer is also improved to more than 120%.
[0058] Figure 6 The experimental results of sparsity and loss function value are schematically shown. According to several settings of the parameters a, b, c, and d in the activation function according to the embodiment of the present disclosure, and the test results when they are applied to the Transformer model for translation tasks. Under the same Transformer model, according to the Figure 6It can be seen that the activation function under different settings has different effects on the forward layer sparsity and the loss function. In the experiment, when a = 0.25, b = 1, c = 0.2, and d = 0.4, the sparsity is increased by 150.95%, and the loss function is reduced to 42.2% of the selected activation function. Both indicators have a large optimization effect.
[0059] Figure 7 The results of the actual text translation task are schematically shown. According to the activation function of the embodiment of the present disclosure, the parameters are set as a = 0.25, b = 1, c = 0.2, and d = 0.4. The German-English translation task of the iwslt14 data set is tested, a total of 160,000 sentences, 10,000 word libraries, and the Transformer model is used as the test network. The results of using the activation function and the training results of using the traditional activation function are compared as shown in Figure 7 The Bleu evaluation of the translation results shows that the score of using the activation function is improved by 0.87 points to 35.16 (the same network, using the traditional activation function, the Bleu score is 34.29 points), and the accuracy is improved by 2.54%.
[0060] Figure 8 The experimental results of the combination of the activation function and the pruning algorithm are schematically shown. According to the activation function of the embodiment of the present disclosure, the parameters are set as a = 0.25, b = 1, c = 0.2, and d = 0.4. Random unstructured pruning is performed on the algorithm to improve the sparsity of the parameters. Different methods are used to prune the weights, and different setting values are used to prune the neural network model, and the influence of the sparsity of the activation function of the embodiment of the present disclosure is as shown in Figure 8 It can be seen from the figure that compared with the L1 norm pruning combined with the activation function of the embodiment of the present disclosure, the random unstructured pruning combined with the activation function of the embodiment of the present disclosure can increase the weight sparsity to 78.26%, and in addition, the activation sparsity of the activation layer is also increased to more than 120%.
[0061] According to the embodiment of the present disclosure, the activation function can improve the algorithm accuracy and the sparsity of the neural network model from the software level, reduce the storage space and the data transmission quantity of the model, and then realize the activation function by using shift and addition operations at a very low resource consumption. In summary, the activation function of the present disclosure applied in the neural network algorithm can improve the algorithm accuracy and the parameter sparsity, and when hardware implementation is required, the activation function circuit module can also be realized at a very low hardware consumption.
[0062] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of a neural network training device provided by an embodiment of the present disclosure. The device can be built-in in an electronic device. The device mainly comprises:
[0063] The acquisition module 910 is configured to acquire an initial training sample data set, and initial training samples in the initial training sample data set include initial text data.
[0064] The preprocessing module 920 is configured to preprocess the initial training samples to generate a training sample data set, and training samples in the training sample data set include text data and label information corresponding to the text data, where the label information is used to represent a translation result of the text data.
[0065] The construction module 930 is configured to construct a neural network to be trained, and the neural network to be trained includes an activation layer, a fully connected forward propagation layer, and a convolutional layer, the activation layer is arranged between the fully connected forward propagation layers or after the convolutional layer, and an activation function of the activation layer is implemented by using addition and shift operations.
[0066] The training module 940 is configured to train the neural network to be trained by using the training sample data set to obtain a trained neural network.
[0067] In an embodiment of the present disclosure, the activation function of the activation layer is max(0, a(x-c), b(x-d)), where x represents an activation value, a < b, the range of a and b is (0, 1), a and b are multiples of 0.125, c < d, and c and d are function zero points.
[0068] In an embodiment of the present disclosure, the device further includes:
[0069] The acquisition module is configured to acquire a value range of parameters a, b, c, and d in the activation function max(0, a(x-c), b(x-d)).
[0070] The setting module is configured to set values of a plurality of activation functions max(0, a(x-c), b(x-d)) according to the value range of a, b, c, and d.
[0071] The model training module is configured to train the trained neural network by using the values of the plurality of activation functions max(0, a(x-c), b(x-d)) to obtain optimal values of the activation function max(0, a(x-c), b(x-d)).
[0072] In an embodiment of the present disclosure, the neural network includes any one of a CNN network model, a Transformer network model, an RNN network model, and an LSTM network model.
[0073] In an embodiment of the present disclosure, after the optimal values of the activation function max(0, a(x-c), b(x-d)) are obtained, a pruning operation is performed.
[0074] According to an embodiment of the present disclosure, program code for execution of the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed completely on a user computing device, partially on a user device, partially on a remote computing device, or completely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).
[0075] Please refer to Figure 10 , Figure 10 A hardware structure diagram of an electronic device is shown.
[0076] The electronic device described in the embodiment includes:
[0077] The memory 41, the processor 42, and the computer program stored on the memory 41 and executable on the processor, and the processor executes the program to realize the foregoing Figure 1 The neural network training method described in the embodiment and Figure 5 The text translation method described in the embodiment.
[0078] Further, the electronic device further includes:
[0079] At least one input device 43; at least one output device 44.
[0080] The above-mentioned memory 41, processor 42, input device 43 and output device 44 are connected through bus 45.
[0081] The input device 43 can be a camera, a touch panel, a physical button or a mouse, etc. The output device 44 can be a display screen.
[0082] The memory 41 can be a high-speed random access memory (RAM, Random Access Memory) memory, or a non-volatile memory such as a disk memory. The memory 41 is used to store a set of executable program codes, and the processor 42 is coupled with the memory 41.
[0083] Further, the embodiments of the present disclosure also provide a computer readable storage medium, which can be arranged in the electronic device in the above-mentioned embodiments, and the computer readable storage medium can be the computer readable storage medium in the above-mentioned embodiments Figure 5 The computer readable storage medium stores a computer program, and the program is executed by the processor to realize the neural network training method and the text translation method described in the above-mentioned embodiments Figure 1 The computer readable storage medium stores a computer program, and the program is executed by the processor to realize the neural network training method and the text translation method described in the above-mentioned embodiments Figure 5 The computer readable storage medium stores a computer program, and the program is executed by the processor to realize the neural network training method and the text translation method described in the above-mentioned embodiments. Further, the computer readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various storage program codes.
[0084] It should be noted that each functional module in each embodiment of the present disclosure can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of a software functional module.
[0085] The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, essentially or in the form of a part or all of the technical solutions of the prior art.
[0086] It should be noted that, for the above-mentioned method embodiments, in order to simplify the description, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.
[0087] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.
[0088] The above is the description of the neural network training method and device, the text translation method, the electronic device and the medium provided by the present application. For those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.
Claims
1. A neural network training method, characterized in that, Including: Obtain an initial training sample dataset, where the initial training samples in the initial training sample dataset include initial text data; Preprocess the initial training samples to generate a training sample dataset. The training samples in the training sample dataset include text data and label information corresponding to the text data. The label information is used to represent the translation result of the text data; Construct a neural network to be trained. The neural network to be trained includes an activation layer, a fully connected forward propagation layer, and a convolutional layer. The activation layer is set between the fully connected forward propagation layers or after the convolutional layer, and the activation function of the activation layer is implemented using addition and shift operations; Use the training sample dataset to train the neural network to be trained to obtain a trained neural network; The activation function of the activation layer is max(0, a(x - c), b(x - d)), where x represents the activation value, a < b, the ranges of a and b are both (0, 1), and both a and b are multiples of 0.125, c < d, and c and d are both function zeros.
2. The neural network training method according to claim 1, characterized in that, Also including: Obtain the value ranges of the parameters a, b, c, and d in the activation function max(0, a(x - c), b(x - d)); According to the value ranges of a, b, c, and d, set the values of multiple activation functions max(0, a(x - c), b(x - d)); Use the values of multiple activation functions max(0, a(x - c), b(x - d)) to train the trained neural network to obtain the optimal value of the activation function max(0, a(x - c), b(x - d)); 3. The neural network training method according to claim 1, characterized in that, The neural network includes any one of a CNN network model, a Transformer network model, a RNN network model, and an LSTM network model.
4. The neural network training method according to claim 1, characterized in that, After obtaining the optimal value of the activation function max(0, a(x - c), b(x - d)), perform a pruning operation.
5. A text translation method, characterized in that, Including: Obtain a dataset of samples to be translated, where the dataset of samples to be translated includes text data to be translated; Input the text data to be translated into the neural network to obtain a translation result of the text data. The neural network is trained by the neural network training method according to any one of claims 1 to 4.
6. A neural network training device, characterized in that, Including: An acquisition module for obtaining an initial training sample dataset, where the initial training samples in the initial training sample dataset include initial text data; A preprocessing module for preprocessing the initial training samples to generate a training sample dataset. The training samples in the training sample dataset include text data and label information corresponding to the text data. The label information is used to represent the translation result of the text data; A construction module for constructing a neural network to be trained. The neural network to be trained includes an activation layer, a fully connected forward propagation layer, and a convolutional layer. The activation layer is set between the fully connected forward propagation layers or after the convolutional layer, and the activation function of the activation layer is implemented using addition and shift operations; A training module for using the training sample dataset to train the neural network to be trained to obtain a trained neural network; The activation function of the activation layer is max(0, a(x - c), b(x - d)), where x represents the activation value, a < b, the ranges of both a and b are (0, 1), and both a and b are multiples of 0.125, c < d, and both c and d are function zeros.
7. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, it implements each step in the neural network training method described in any one of claims 1 to 4, or implements each step in the text translation method described in claim 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step in the neural network training method described in any one of claims 1 to 4, or implements each step in the text translation method described in claim 5.