A method and apparatus for generating an RNN model

Abstract

This application discloses a method and apparatus for generating RNN models to address the technical problem of low inference speed in existing RNN models due to frequent quantization and dequantization operations. The method includes: generating a first RNN model, which includes a matrix addition function and an activation function; the result of the matrix addition function is quantized and then input into the activation function; the result of the activation function is dequantized and then used as the output matrix; obtaining the accuracy loss of the first RNN model during training; if the accuracy loss is not less than a preset threshold, updating the weight matrix of the first RNN model based on the accuracy loss to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during training is less than the preset threshold; and generating a second RNN model, which is the updated first RNN model after removing the quantization and dequantization operations.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a method and apparatus for generating RNN models. Background Technology

[0002] Recurrent neural networks (RNNs) are a widely used network architecture in deep learning, primarily for processing time-dependent features, such as automatic speech recognition (ASR) and semantic understanding. Major variants include long short-term memory networks (LSTM), bidirectional RNNs (Bi-RNNs), and gated recurrent units (GRUs).

[0003] During training, RNN models can use either fixed-point or floating-point computation for operators. Since fixed-point computation reduces precision, RNN models using only fixed-point computation suffer significant precision loss. Furthermore, the limited computational resources at the terminal end result in slow inference speeds for RNN models using only floating-point computation. Therefore, 8-bit weight quantization is commonly used to improve RNN models where all operators use floating-point computation. Specifically, 8-bit weight quantization is used to generate the RNN model's input online. This involves first quantizing the input value from floating-point to fixed-point, then performing matrix multiplication, and finally dequantizing the result from fixed-point to floating-point. By replacing some floating-point computations with fixed-point computations, the computational load is reduced while the inference speed of the RNN model is improved.

[0004] However, since the convolution operation of the RNN model is a circular convolution and the input is a sequential input, after replacing the floating-point calculation of some operators of the RNN model with fixed-point calculation, each input will undergo quantization and dequantization operations. Although the speed of fixed-point matrix multiplication after quantization is improved compared with floating-point matrix multiplication, the frequent quantization and dequantization operations increase the computational complexity and random access memory (RAM) overhead, resulting in the low inference speed of the RNN model. Summary of the Invention

[0005] This application provides a method and apparatus for generating RNN models to solve the technical problem that existing RNN models have low inference speed due to frequent quantization and dequantization operations.

[0006] In a first aspect, embodiments of this application provide a method for generating an RNN model, comprising:

[0007] A first recurrent neural network (RNN) model is generated; wherein the first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any of the matrix multiplication functions. The calculation result of any of the matrix multiplication functions is dequantized and then input into the matrix addition function. The calculation result of the matrix addition function is quantized and then input into the activation function. The calculation result of the activation function is dequantized and then used as the output matrix.

[0008] Input the floating-point type of the input matrix into the first RNN model to train the model, and obtain the accuracy loss of the first RNN model during model training.

[0009] If the accuracy loss is not less than a preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold.

[0010] Generate a second RNN model; wherein the second RNN model is the updated first RNN model after removing quantization and dequantization processing.

[0011] Based on the above technical solution, a first RNN model can be generated first. The first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any matrix multiplication function. The calculation result of any matrix multiplication function is dequantized and then input into the matrix addition function. The calculation result of the matrix addition function is quantized and then input into the activation function. The calculation result of the activation function is dequantized and then used as the output matrix. Then, the input matrix with a floating-point numerical type is input into the first RNN model for model training. The accuracy loss of the first RNN model during model training is obtained. If the accuracy loss is not less than a preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain an updated first RNN model. This process continues until the accuracy loss of the updated first RNN model during model training is less than the preset threshold. Then, a second RNN model is generated. The second RNN model is the updated first RNN model after removing the quantization and dequantization processes. By updating the weight matrix of the first RNN model, an updated first RNN model is obtained, which makes the accuracy loss of the updated first RNN model during model training less than a preset threshold. Then, by removing the additional quantization and dequantization processing in the updated first RNN model, a second RNN model is obtained, which allows the data flow of the second RNN model during model inference to be fully fixed-point, reducing computational complexity and RAM overhead. This improves the inference speed of the RNN model while ensuring low accuracy loss.

[0012] In one possible design, updating the weight matrix of the first RNN model based on the accuracy loss to obtain the updated first RNN model includes:

[0013] Obtain the gradient corresponding to any weight in the weight matrix of the first RNN model and the accuracy loss; wherein the gradient is the derivative of the accuracy loss with respect to any weight;

[0014] The updated weight is obtained by updating any weight based on the gradient; wherein the updated weight is the sum of the gradient and any weight before the update.

[0015] In one possible design, the activation function includes an inverse quantization function, a linear activation function, and a quantization function;

[0016] The calculation formula for the activation function is as follows:

[0017] X out =Q(Linear_ACT(D e (X in )))

[0018] Among them, D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in This is the calculation result of the matrix addition function after quantization.

[0019] Based on the above technical solution, the activation function can include inverse quantization function, linear activation function, and quantization function. The specific formula for calculating the activation function is: X out =Q(Linear_ACT(D e (X in ))), where D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in X is the result of the matrix addition function calculation after quantization. out This represents the calculation result of the activation function. By replacing the nonlinear activation function in the existing RNN model with a linear activation function, and connecting the dequantization function and quantization function to the input and output of the linear activation function respectively, the precision drift that exists in the existing RNN model when both the input and output values ​​are fixed-point types is avoided.

[0020] In one possible design, generating the second RNN model includes:

[0021] Obtain a first quantization factor and a second quantization factor when training the updated first RNN model; wherein, the first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function, and the second quantization factor is the quantization factor of the dequantization process of the calculation result of any matrix multiplication function before inputting the matrix addition function.

[0022] The formula for calculating the matrix multiplication function of the second RNN model is determined based on the first quantization factor and the second quantization factor.

[0023] Obtain the third quantization factor and the fourth quantization factor when training the updated first RNN model; wherein, the third quantization factor is the quantization factor of the calculation result of the matrix addition function before it is input into the activation function, and the fourth quantization factor is the quantization factor of the calculation result of the activation function before it is used as the output matrix.

[0024] The quantization factor of the activation function of the second RNN model is determined based on the third quantization factor and the fourth quantization factor.

[0025] Remove all quantization and dequantization processes from the updated first RNN model, except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, to generate the second RNN model.

[0026] Based on the above technical solution, the first and second quantization factors of the updated first RNN model during model training can be obtained first. The first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function. The second quantization factor is the quantization factor of the calculation result of any matrix multiplication function before inputting the matrix addition function. The calculation formula of the matrix multiplication function of the second RNN model is determined based on the first and second quantization factors. Then, the third and fourth quantization factors of the updated first RNN model during model training are obtained. The third quantization factor is the quantization factor of the calculation result of the matrix addition function before inputting the activation function. The fourth quantization factor is the quantization factor of the calculation result of the activation function before being used as the output matrix. The quantization factor of the activation function of the second RNN model is determined based on the third and fourth quantization factors. Finally, all quantization and dequantization processes in the updated first RNN model, except for the quantization processes performed by the quantization functions included in the activation function and the dequantization processes performed by the dequantization functions, are removed to generate the second RNN model. By obtaining the quantization factors of each quantization and dequantization process during the training of the updated first RNN model, the calculation formula of the matrix multiplication function and the quantization factor of the activation function of the second RNN model are determined. After removing the additional quantization and dequantization processes in the updated first RNN model, the data flow of the second RNN model during model inference can be fully fixed-point, reducing computational complexity and RAM overhead. This improves the inference speed of the RNN model while ensuring low accuracy loss.

[0027] In one possible design, determining the matrix multiplication function of the second RNN model based on the first quantization factor and the second quantization factor includes:

[0028] The formula for calculating the matrix multiplication function of the second RNN model is as follows:

[0029]

[0030] Where Scale_w is the scaling factor of the weight matrix in the first quantization factor, zero_point_w is the zero point of the weight matrix in the first quantization factor, Scale_x is the scaling factor of the input matrix in the first quantization factor, zero_point_x is the zero point of the input matrix in the first quantization factor, and Scale_Y wx zero_point_Y is the scaling factor for the matrix multiplication result in the second quantization factor. wx W is the zero point of the matrix multiplication result in the second quantization factor. q X is a weight matrix of fixed-point numerical type. q Given an input matrix of fixed-point numerical type, Y wxq This is the result of a matrix multiplication with a fixed-point numerical type.

[0031] In one possible design, determining the quantization factor of the activation function of the second RNN model based on the third quantization factor and the fourth quantization factor includes:

[0032] The third quantization factor is used to determine the quantization factor of the dequantization process performed by the dequantization function included in the activation function of the second RNN model.

[0033] The quantization factor is determined based on the fourth quantization factor, which is the quantization factor of the quantization function included in the activation function of the second RNN model.

[0034] Secondly, this application also provides an apparatus for generating an RNN model. The apparatus has the functionality to implement the first aspect or any possible design method described in the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functionality, such as a first generation module, a training module, an update module, and a second generation module.

[0035] A first generation module is used to generate a first recurrent neural network (RNN) model. The first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any of the matrix multiplication functions. The calculation result of any of the matrix multiplication functions is dequantized and then input into the matrix addition function. The calculation result of the matrix addition function is quantized and then input into the activation function. The calculation result of the activation function is dequantized and then used as the output matrix.

[0036] The training module is used to input the input matrix of floating-point type into the first RNN model for model training and to obtain the accuracy loss of the first RNN model during model training.

[0037] An update module is used to update the weight matrix of the first RNN model based on the accuracy loss if the accuracy loss is not less than a preset threshold, so as to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold.

[0038] The second generation module is used to generate a second RNN model; wherein the second RNN model is the updated first RNN model after removing quantization and dequantization processing.

[0039] In one possible design, the updated model is specifically used for:

[0040] Obtain the gradient corresponding to any weight in the weight matrix of the first RNN model and the accuracy loss; wherein the gradient is the derivative of the accuracy loss with respect to any weight;

[0041] The updated weight is obtained by updating any weight based on the gradient; wherein the updated weight is the sum of the gradient and any weight before the update.

[0042] In one possible design, the activation function includes an inverse quantization function, a linear activation function, and a quantization function;

[0043] The calculation formula for the activation function is as follows:

[0044] X out =Q(Linear_ACT(D e (X in )))

[0045] Among them, D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in This is the calculation result of the matrix addition function after quantization.

[0046] In one possible design, the second generation module is specifically used for:

[0047] Obtain a first quantization factor and a second quantization factor when training the updated first RNN model; wherein, the first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function, and the second quantization factor is the quantization factor of the dequantization process of the calculation result of any matrix multiplication function before inputting the matrix addition function.

[0048] The formula for calculating the matrix multiplication function of the second RNN model is determined based on the first quantization factor and the second quantization factor.

[0049] Obtain the third quantization factor and the fourth quantization factor when training the updated first RNN model; wherein, the third quantization factor is the quantization factor of the calculation result of the matrix addition function before it is input into the activation function, and the fourth quantization factor is the quantization factor of the calculation result of the activation function before it is used as the output matrix.

[0050] The quantization factor of the activation function of the second RNN model is determined based on the third quantization factor and the fourth quantization factor.

[0051] Remove all quantization and dequantization processes from the updated first RNN model, except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, to generate the second RNN model.

[0052] In one possible design, the second generation module is specifically used for:

[0053] The formula for calculating the matrix multiplication function of the second RNN model is as follows:

[0054]

[0055] Where Scale_w is the scaling factor of the weight matrix in the first quantization factor, zero_point_w is the zero point of the weight matrix in the first quantization factor, Scale_x is the scaling factor of the input matrix in the first quantization factor, zero_point_x is the zero point of the input matrix in the first quantization factor, and Scale_Y wx zero_point_Y is the scaling factor for the matrix multiplication result in the second quantization factor. wx W is the zero point of the matrix multiplication result in the second quantization factor. q X is a weight matrix of fixed-point numerical type. q Given an input matrix of fixed-point numerical type, Y wxq This is the result of a matrix multiplication with a fixed-point numerical type.

[0056] In one possible design, the second generation module is specifically used for:

[0057] The third quantization factor is used to determine the quantization factor of the dequantization process performed by the dequantization function included in the activation function of the second RNN model.

[0058] The quantization factor is determined based on the fourth quantization factor, which is the quantization factor of the quantization function included in the activation function of the second RNN model.

[0059] Thirdly, this application also provides an electronic device, the electronic device including at least one processor and a memory; the memory stores one or more computer programs; when the one or more computer programs stored in the memory are executed by the at least one processor, the electronic device performs the method described in the first aspect or any possible design of the first aspect.

[0060] Fourthly, this application also provides a computer storage medium including computer instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible design of the first aspect.

[0061] Fifthly, this application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect or any possible design of the first aspect.

[0062] In a sixth aspect, this application also provides a chip that can be coupled to the memory of an electronic device for calling a computer program stored in the memory and executing the method described in the first aspect or any possible design of the first aspect.

[0063] The beneficial effects of the second to fifth aspects and any of their possible designs can be referred to the description of the beneficial effects of the methods in the first aspect and any of their possible designs. Attached Figure Description

[0064] Figure 1 A schematic diagram of an existing RNN model provided for an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0066] Figure 3 A flowchart illustrating a method for generating an RNN model provided in an embodiment of this application;

[0067] Figure 4a A schematic diagram of a first RNN model provided in an embodiment of this application;

[0068] Figure 4b A schematic diagram of a second RNN model provided in an embodiment of this application;

[0069] Figure 5A schematic diagram of the structure of an apparatus for generating an RNN model provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0072] like Figure 1 The diagram shown is a schematic of an existing RNN model provided in an embodiment of this application. Figure 1 The RNN model in this example is an improved version of the RNN model that uses 8-bit weight quantization to perform floating-point calculations on all operators. This RNN model includes two matrix multiplication functions (Mult(·)), one matrix addition function (Add(·)), and one activation function (ACT(·)). ACT(·) can be either a Sigmoid function or a Tanh function. W and H represent the two weight matrices of the RNN model, and X and R represent the two input matrices. The weight matrices W and H are of fixed-point type (int8) with a value range of [-128, 127], for example, 99 or 88. The input matrices X and R are of floating-point type (float32), for example, 0.05857592 or 0.1095115. The process of training or inferring this RNN model is as follows:

[0073] (1) Input matrices X and R are respectively input into the corresponding quantization function Q(·) for quantization processing, so that the numerical type of each scalar value of input matrices X and R is quantized from floating-point type float32 to fixed-point type int8. For example, after input matrix X is input into the corresponding Q(·), the process of Q(·) quantizing input matrix X is as follows: Q(·) first records the maximum and minimum values ​​of input matrix X as X min and X max Then calculate the quantization factor of the input matrix X: scaling factor Scale_x = (X max -X min ) / (127+128), and zero point x = 127-X max / Scale_x, where Scale_x represents the mapping between floating-point and fixed-point values, and zero_point_x describes the value of zero after quantization. Assume that any scalar value of the input matrix X is X. fi Then its quantized value is X qi=X fi / Scale_x+zero_point_x. Perform the above operation on each scalar value of the input matrix X, thereby quantizing the numeric type of each scalar value of the input matrix X from the floating-point type float32 to the fixed-point type int8. Similarly, the numeric type of each scalar value of the input matrix R can be quantized from the floating-point type float32 to the fixed-point type int8.

[0074] (2) Perform matrix multiplication between the quantized input matrix X and the corresponding Mult(·) and the weight matrix W, and perform matrix multiplication between the quantized input matrix R and the corresponding Mult(·) and the weight matrix H, to obtain the matrix multiplication result Y of the quantized input matrix X and the weight matrix W. XW The matrix multiplication result Y of the quantized input matrix R and the weight matrix H is also calculated. RH , where Y XW and Y RH The numeric type is fixed-point type int32, and the numeric range is [-2147483648, 2147483647];

[0075] (3) Multiply the two matrices and calculate the result Y. XW and Y RH Input the corresponding dequantization function De(·) to perform dequantization processing, so that the result of the matrix multiplication Y is... XW and Y RH The numeric type of each scalar value is dequantized from fixed-point type int32 to floating-point type float32. For example, the matrix multiplication result Y of the quantized input matrix X and the weight matrix W is calculated as follows: XW After inputting the corresponding De(·), the matrix multiplication result Y of De(·) is calculated. XW The process of inverse quantization is as follows: De(·) first records the matrix multiplication result Y. XW The maximum and minimum values ​​are Y xwmin and Y xwmax Then calculate the matrix multiplication result Y. XW Quantization factor: Scaling factor Scale_Y xw =(Y xwmax -Y xwmin ) / (2147483647+2147483648), and zero point_Y wx =2147483647-Y xwmax / Scale_Y xw Assuming the matrix multiplication result is Y XW Any scalar value of Y wxqi =X qi *Wi , where X qi Let W be any scalar value of the quantized weight matrix X. i Let Y be any scalar value of the weight matrix W, then its dequantized value is Y. wxfi =(Y wxqi -zero_point_Y wx )*Scale_Y xw The result of matrix multiplication, Y XW Perform the above operation on each scalar value to multiply the matrix by the result Y. XW The numeric type of each scalar value is dequantized from the fixed-point type int32 to the floating-point type float32. Similarly, the matrix multiplication result Y can be calculated. RH The numeric type of each scalar value is dequantized from the fixed-point type int32 to the floating-point type float32;

[0076] (4) Multiply the matrix after dequantization by the calculated result Y XW The matrix multiplication result Y after dequantization RH Input Add(·) to perform matrix addition calculation, obtain the matrix addition result Y, and then input the matrix addition result Y into ACT(·) to perform activation calculation, obtain the output matrix of the RNN model, where the output matrix of the RNN model can be used as the input matrix R for the next model training or model inference.

[0077] However, because the convolution operation of the RNN model is a circular convolution and the input is a temporal input, it makes... Figure 1 In the RNN model, each input undergoes quantization and dequantization. Although quantization improves the speed of fixed-point matrix multiplication compared to floating-point matrix multiplication, frequent quantization and dequantization will lead to… Figure 1 The inference speed of the RNN model in the text is relatively low.

[0078] In view of this, embodiments of this application provide a method for generating an RNN model. This method first generates a first RNN model, which includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any matrix multiplication function. The result of the matrix multiplication function is dequantized and then input into the matrix addition function. The result of the matrix addition function is quantized and then input into the activation function. The result of the activation function is dequantized and then used as the output matrix. Then, the input matrix, whose numerical type is floating-point, is input into the first RNN model for model training. The accuracy loss of the first RNN model during training is obtained. If the accuracy loss is not less than a preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain an updated first RNN model. This process continues until the accuracy loss of the updated first RNN model during training is less than the preset threshold. Finally, a second RNN model is generated, where the second RNN model is the updated first RNN model after removing the quantization and dequantization processes. By updating the weight matrix of the first RNN model, an updated first RNN model is obtained, which makes the accuracy loss of the updated first RNN model during model training less than a preset threshold. Then, by removing the additional quantization and dequantization processes in the updated first RNN model, a second RNN model is obtained, which allows the data flow of the second RNN model during model inference to be fully fixed-point, reducing computational complexity and random access memory (RAM) overhead. This improves the inference speed of the RNN model while ensuring low accuracy loss.

[0079] It should be understood that the technical solutions in the embodiments of this application can be applied to any electronic device. For example, it can be a portable electronic device that includes functions such as a personal digital assistant and / or a music player, such as a mobile phone, a tablet computer, or a wearable device with wireless communication capabilities (e.g., a smartwatch). Exemplary embodiments of portable electronic devices include, but are not limited to, those equipped with... Alternatively, it could be a portable electronic device with another operating system. The aforementioned portable electronic device could also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, the aforementioned electronic device may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0080] The following uses a mobile phone as an example to briefly describe the structure of the electronic device to which this application embodiment can be applied, so as to facilitate understanding of the technical solution by those skilled in the art. For example, as shown... Figure 2 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 2The mobile phone 100 shown may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0081] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the mobile phone 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 may also include memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or is recurring. If the processor 110 needs to reuse the instruction or data, it can directly retrieve it from the memory. This avoids repeated access, reduces the processor 110's waiting time, and thus improves system efficiency.

[0082] USB interface 130 is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge mobile phone 100, and can also be used for data transfer between mobile phone 100 and peripheral devices. Charging management module 140 receives charging input from the charger. Power management module 141 connects to battery 142, charging management module 140, and processor 110. Power management module 141 receives input from battery 142 and / or charging management module 140, providing power to processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0083] The wireless communication function of mobile phone 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0084] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on the mobile phone 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0085] The wireless communication module 160 can provide solutions for wireless communication applications on the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0086] In some embodiments, antenna 1 of mobile phone 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling mobile phone 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0087] In some embodiments, in addition to GNSS as described above, mobile phone 100 may also use wireless fidelity (WiFi) positioning or radio frequency identification (RFID) positioning to determine the location of the terminal device.

[0088] The display screen 194 is used to display the application's interface, such as a page showing the movement trajectory determined based on the user's motion data collected by the mobile phone 100. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0089] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0090] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and software code for at least one application. The data storage area may store data generated during the use of the mobile phone 100 (e.g., captured images, recorded videos, etc.). Furthermore, the internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0091] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, images, videos, and other files can be saved on the external storage card.

[0092] The mobile phone 100 can achieve audio functions such as music playback and recording through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0093] The sensor module 180 may include a pressure sensor 180A, an acceleration sensor 180B, a touch sensor 180C, a height sensor 180D, etc.

[0094] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194.

[0095] Touch sensor 180C, also known as a "touch panel," can be located on display screen 194. The touch sensor 180C and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180C detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180C may also be located on the surface of mobile phone 100, in a different position than display screen 194.

[0096] Altitude sensor 180D is used to collect altitude and barometric pressure data and send the collected altitude and barometric pressure data to processor 110 so that processor 110 can calculate the altitude information of mobile phone 100 from the altitude and barometric pressure data.

[0097] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The mobile phone 100 can receive button inputs and generate key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to different applications (such as taking photos, playing audio, etc.). Touch vibration feedback effects can also be customized. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile phone 100.

[0098] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Furthermore, Figure 2 The combination or connection relationship between the components can also be adjusted and modified.

[0099] The above describes the electronic devices to which the technical solutions in the embodiments of this application can be applied. The following describes a method for generating an RNN model provided by the embodiments of this application, in conjunction with the accompanying drawings.

[0100] It should be understood that "at least one" in the embodiments of this application refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c.

[0101] Furthermore, unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria, and do not indicate that the content, priority, or importance of these two criteria are different.

[0102] like Figure 3 The diagram shown is a flowchart illustrating a method for generating an RNN model according to an embodiment of this application. This method for generating an RNN model can be applied to the above-mentioned... Figure 2 The shown or the Figure 2 On electronic devices with similar functional structures, the method for generating RNN models is executed by various hardware components within the electronic device in conjunction with corresponding software programs. The specific process of generating RNN models is described below.

[0103] S301, Generate the first RNN model.

[0104] In some embodiments, a first RNN model can be generated, comprising two matrix multiplication functions, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any matrix multiplication function. The result of the matrix multiplication function is dequantized and then input into the matrix addition function. The result of the matrix addition function is quantized and then input into the activation function. The result of the activation function is dequantized and then used as the output matrix. The output matrix can be used as the input matrix for the next training or inference iteration of the first RNN model.

[0105] It should be noted that, in the embodiments of this application, the activation function may include an inverse quantization function, a linear activation function, and a quantization function, and the specific calculation formula for the activation function is: X out =Q(Linear_ACT(D e (X in ))), where D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in X is the result of the matrix addition function calculation after quantization. out This represents the calculation result of the activation function. By replacing the nonlinear activation function in the existing RNN model with a linear activation function, such as replacing the Sigmoi function with the Hard_Sigmoid function and the Tanh function with the Hard_Tanh function, and by connecting the dequantization function and the quantization function to the input and output of the linear activation function respectively, the precision drift that exists in the existing RNN model when the nonlinear activation function has fixed-point numerical types in both input and output is avoided.

[0106] For example, such as Figure 4a The diagram shown is a schematic representation of a first RNN model provided in an embodiment of this application. Figure 4aThe first RNN model in the code includes two matrix multiplication functions Mult(·), a matrix addition function Add(·), and an activation function Q(Linear_ACT(D)). e (·))), where Linear_ACT(·) can be either the Hard_Sigmoid function or the Hard_Tanh function, W f and H f X and R represent the two weight matrices of the first RNN model, respectively. The weight matrix W, input matrix X, weight matrix H, and input matrix R are all of floating-point type. The training process of the first RNN model is as follows: The weight matrix W, input matrix X, weight matrix H, and input matrix R are quantized by their respective quantization functions Q(·), then input to the corresponding matrix multiplication function Mult(·). The result of the two matrix multiplication functions Mult(·) is dequantized by the corresponding dequantization function De(·), then input to the matrix addition function Add(·). The result of the matrix addition function Add(·) is quantized by the corresponding quantization function Q(·), then input to the activation function Q(Linear_ACT(D)). e (·))), activation function Q(Linear_ACT(D e The calculation result of (·))) is used as the output matrix R after being dequantized by the corresponding dequantization function De(·). f Output matrix R f It can be used as the input matrix R for the next training iteration of the first RNN model. f .

[0107] S302. Input the input matrix of floating-point type into the first RNN model for model training, and obtain the accuracy loss of the first RNN model during model training.

[0108] In some embodiments, after generating the first RNN model, an input matrix of floating-point type can be input into the first RNN model for model training, thereby obtaining the accuracy loss during model training. When the first RNN model is used for automatic speech recognition or semantic understanding, the input matrix can be different speech types.

[0109] For example, if the first RNN model is Figure 4a The first RNN model in the example takes the speech "I love Shanghai" as its input matrix. It then uses Fourier transform and a windowing function to segment the input matrix into four frames: "I", "love", "Shanghai", and "Hai". At time 1, "I" is used as the input matrix X, which is a floating-point number. fThe empty character "" is used as the input matrix R of floating-point type. f At 2 o'clock, "love" is X f "I" as R f And so on. X f and R f The process of training the first RNN model is as follows:

[0110] (1) Input matrix X f and input matrix R f Input the corresponding quantization function Q(·) for quantization processing respectively:

[0111]

[0112]

[0113] Similarly, the weight matrix W f and weight matrix H f Input the corresponding quantization function Q(·) for quantization processing respectively:

[0114]

[0115]

[0116] This makes the weight matrix W f Input matrix X f Weight matrix H f and input matrix R f The numeric type of each scalar value is quantized from the floating-point type float32 to the fixed-point type int8, with a numeric range of [-128, 127].

[0117] (2) Quantize the input matrix X q Input the corresponding Mult(·) and the quantized weight matrix W q Perform matrix multiplication, and quantize the input matrix R. q Input the corresponding Mult(·) and the quantized weight matrix H q Perform matrix multiplication to obtain the quantized input matrix X. q With the quantized weight matrix W q The matrix multiplication result Y XWq and the quantized input matrix R q With the quantized weight matrix H q The matrix multiplication result Y RHq , where Y XWq and Y RHqThe numeric type is fixed-point type int32, and the numeric range is [-2147483648, 2147483647];

[0118] (3) Multiply the two matrices and calculate the result Y. XWq and Y RHq Input the corresponding dequantization function De(·) to perform dequantization processing, so that the result of the matrix multiplication Y is... XWq and Y RHq The numeric type of each scalar value is dequantized from the fixed-point type int32 to the floating-point type float32;

[0119] (4) First multiply the matrix after dequantization by the calculated result Y. XWf The matrix multiplication result Y after dequantization RHf Input Add(·) to perform matrix addition and get the matrix addition result Y. f Then add the matrix to the calculated result Y f Input the corresponding quantization function Q(·) for quantization processing, so that the matrix addition calculation result Y f Each scalar value is quantized from a floating-point type (float32) to a fixed-point type (int8), and then the quantized matrix is ​​added to the calculated result Y. q Input Q(Linear_ACT(D) e Activation calculations are performed using (·))) to obtain the activation result T. q And activate the calculation result T q Input the corresponding dequantization function De(·) to perform dequantization processing, which dequantizes the numeric type of each scalar value in the activation calculation result from fixed-point type int8 to floating-point type float32, thereby dequantizing the activation calculation result T. f R, the output matrix of the first RNN model f The output matrix R of the first RNN model f It can be used as the input matrix R for the next model training. f .

[0120] S303. If the accuracy loss is not less than the preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain the updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold.

[0121] In some embodiments, after inputting an input matrix of floating-point type into the first RNN model for model training and obtaining the accuracy loss of the first RNN model during model training, it can be determined whether the accuracy loss is not less than a preset threshold. If the accuracy loss is not less than the preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain the updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold.

[0122] Specifically, the accuracy loss and the gradient corresponding to any weight in the weight matrix of the first RNN model are obtained, where the gradient is the derivative of the accuracy loss with respect to any weight. Based on the gradient, any weight is updated to obtain the updated weight, where the updated weight is the sum of the gradient and the weight before the update. This process is repeated until every weight in every weight matrix of the first RNN model has been updated, thereby reducing the error between the output label and the true label of the first RNN model.

[0123] S304. Generate the second RNN model.

[0124] In some embodiments, after the accuracy loss during training of the updated first RNN model is less than a preset threshold, a second RNN model can be generated. The second RNN model is the updated first RNN model after removing quantization and dequantization processes. Specifically, the process of generating the second RNN model is as follows:

[0125] (1) Obtain the first quantization factor and the second quantization factor when training the updated first RNN model. The first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function. The second quantization factor is the quantization factor of the calculation result of any matrix multiplication function before input matrix addition function.

[0126] (2) Determine the formula for calculating the matrix multiplication function of the second RNN model based on the first and second quantization factors:

[0127]

[0128] Where Scale_w is the scaling factor of the weight matrix in the first quantization factor, zero_point_w is the zero point of the weight matrix in the first quantization factor, Scale_x is the scaling factor of the input matrix in the first quantization factor, zero_point_x is the zero point of the input matrix in the first quantization factor, and Scale_Y wx zero_point_Y is the scaling factor for the matrix multiplication result in the second quantization factor.wx W is the zero point of the matrix multiplication result in the second quantization factor. q X is a weight matrix of fixed-point numerical type. q Given an input matrix of fixed-point numerical type, Y wxq This is the result of a matrix multiplication with a numeric type of fixed point.

[0129] (3) Obtain the third and fourth quantization factors when training the updated first RNN model. The third quantization factor is the quantization factor of the matrix addition function calculation result before inputting it into the activation function, and the fourth quantization factor is the quantization factor of the activation function calculation result before being used as the output matrix.

[0130] (4) Determine the quantization factor of the activation function of the second RNN model based on the third quantization factor and the fourth quantization factor. For example, determine the quantization factor of the inverse quantization process of the activation function of the second RNN model based on the third quantization factor, and determine the quantization factor of the quantization process of the quantization function of the activation function of the second RNN model based on the fourth quantization factor.

[0131] (5) Remove all quantization and dequantization processes from the updated first RNN model except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, and generate the second RNN model.

[0132] For example, such as Figure 4b The diagram shown is a schematic representation of a second RNN model provided in an embodiment of this application. Figure 4b The second RNN model in the code includes two matrix multiplication functions Mult(·), a matrix addition function Add(·), and an activation function Q(Linear_ACT(D)). e (·))), where Linear_ACT(·) can be either the Hard_Sigmoid function or the Hard_Tanh function, W q and H q These represent the two weight matrices of the second RNN model, X and X respectively. q and R q These represent the two input matrices of the second RNN model, and the weight matrix W. q Input matrix X q Weight matrix H q and input matrix R q The numerical data types are all fixed-point types. The inference process of the second RNN model is as follows: weight matrix W q Input matrix X q Weight matrix H q and input matrix Rq Input the corresponding matrix multiplication function Mult(·) respectively. Input the result of the two matrix multiplication functions Mult(·) into the matrix addition function Add(·). Input the result of the matrix addition function Add(·) into the activation function Q(Linear_ACT(D e (·))), activation function Q(Linear_ACT(D e The calculation result of (·))) is used as the output matrix R. f Output matrix R f It can be used as the input matrix R for the next inference iteration of the second RNN model. f By removing additional quantization and dequantization processes, the data flow of the second RNN model during inference can be fully fixed-point, reducing computational complexity and RAM overhead, and improving the inference speed of the RNN model.

[0133] Based on the above technical solution, a first RNN model can be generated first. This first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any matrix multiplication function. The result of the matrix multiplication function is dequantized and then input into the matrix addition function. The result of the matrix addition function is quantized and then input into the activation function. The result of the activation function is dequantized and then used as the output matrix. The activation function can include a dequantization function, a linear activation function, and a quantization function. The specific formula for calculating the activation function is: X out =Q(Linear_ACT(D e (X in ))), where D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in X is the result of the matrix addition function calculation after quantization. outThe result is the calculation result of the activation function. By replacing the non-linear activation function in the existing RNN model with a linear activation function, such as replacing the Sigmoi function with the Hard_Sigmoid function and the Tanh function with the Hard_Tanh function, and adding dequantization and quantization functions to the input and output of the linear activation function respectively, the precision drift that exists in the existing RNN model when both the input and output values ​​are fixed-point types is avoided. Then, the input matrix with floating-point values ​​is input into the first RNN model for training. The precision loss of the first RNN model during training is obtained. If the precision loss is not less than a preset threshold, the weight matrix of the first RNN model is updated based on the precision loss to obtain an updated first RNN model. This process continues until the precision loss of the updated first RNN model during training is less than the preset threshold, at which point a second RNN model is generated. The second RNN model is the updated first RNN model after removing quantization and dequantization processing. By updating the weight matrix of the first RNN model, an updated first RNN model is obtained, which makes the accuracy loss of the updated first RNN model during model training less than a preset threshold. Then, by removing the additional quantization and dequantization processes in the updated first RNN model, a second RNN model is obtained, which allows the data flow of the second RNN model during model inference to be fully fixed-point, reducing computational complexity and random access memory (RAM) overhead. This improves the inference speed of the RNN model while ensuring low accuracy loss.

[0134] It should be understood that in the embodiments of this application, the electronic device may perform some or all of the steps in the embodiments of this application, but these steps are only examples, and the electronic device may also perform other steps or variations of various steps. The steps may be performed in different orders as presented in the embodiments of this application, and it is not necessary to perform all the steps in the embodiments of this application.

[0135] Furthermore, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0136] It should be understood that, in the embodiments of this application, the electronic device can be divided into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one functional module. The integrated module can be implemented in the form of a hardware structure or a software module. It should be noted that the module division in the embodiments of this application is illustrative and is only a logical functional division; in actual implementation, there may be other division methods.

[0137] Based on the above embodiments, this application also provides an apparatus for generating an RNN model, the apparatus being used to implement the method for generating an RNN model as shown in the above figures, such as... Figure 5 The diagram shown is a structural schematic of an apparatus for generating an RNN model according to an embodiment of this application. The apparatus 500 may include:

[0138] The first generation module 501 is used to generate a first recurrent neural network (RNN) model. The first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any of the matrix multiplication functions. The calculation result of any of the matrix multiplication functions is dequantized and then input into the matrix addition function. The calculation result of the matrix addition function is quantized and then input into the activation function. The calculation result of the activation function is dequantized and then used as the output matrix.

[0139] Training module 502 is used to input an input matrix of floating-point type into the first RNN model for model training and to obtain the accuracy loss of the first RNN model during model training.

[0140] The update module 503 is used to update the weight matrix of the first RNN model based on the accuracy loss if the accuracy loss is not less than a preset threshold, so as to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold.

[0141] The second generation module 504 is used to generate a second RNN model; wherein the second RNN model is the updated first RNN model after removing quantization and dequantization processing.

[0142] In one possible design, the updated model 503 is specifically used for:

[0143] Obtain the gradient corresponding to any weight in the weight matrix of the first RNN model and the accuracy loss; wherein the gradient is the derivative of the accuracy loss with respect to any weight;

[0144] The updated weight is obtained by updating any weight based on the gradient; wherein the updated weight is the sum of the gradient and any weight before the update.

[0145] In one possible design, the activation function includes an inverse quantization function, a linear activation function, and a quantization function;

[0146] The calculation formula for the activation function is as follows:

[0147] X out =Q(Linear_ACT(D e (X in )))

[0148] Among them, D e (·) is the dequantization function, Linear_ACT(·) is the linear activation function, Q(·) is the quantization function, and X in This is the calculation result of the matrix addition function after quantization.

[0149] In one possible design, the second generation module 504 is specifically used for:

[0150] Obtain a first quantization factor and a second quantization factor when training the updated first RNN model; wherein, the first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function, and the second quantization factor is the quantization factor of the dequantization process of the calculation result of any matrix multiplication function before inputting the matrix addition function.

[0151] The formula for calculating the matrix multiplication function of the second RNN model is determined based on the first quantization factor and the second quantization factor.

[0152] Obtain the third quantization factor and the fourth quantization factor when training the updated first RNN model; wherein, the third quantization factor is the quantization factor of the calculation result of the matrix addition function before it is input into the activation function, and the fourth quantization factor is the quantization factor of the calculation result of the activation function before it is used as the output matrix.

[0153] The quantization factor of the activation function of the second RNN model is determined based on the third quantization factor and the fourth quantization factor.

[0154] Remove all quantization and dequantization processes from the updated first RNN model, except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, to generate the second RNN model.

[0155] In one possible design, the second generation module 504 is specifically used for:

[0156] The formula for calculating the matrix multiplication function of the second RNN model is as follows:

[0157]

[0158] Where Scale_w is the scaling factor of the weight matrix in the first quantization factor, zero_point_w is the zero point of the weight matrix in the first quantization factor, Scale_x is the scaling factor of the input matrix in the first quantization factor, zero_point_x is the zero point of the input matrix in the first quantization factor, and Scale_Y wx zero_point_Y is the scaling factor for the matrix multiplication result in the second quantization factor. wx W is the zero point of the matrix multiplication result in the second quantization factor. q X is a weight matrix of fixed-point numerical type. q Given an input matrix of fixed-point numerical type, Y wxq This is the result of a matrix multiplication with a fixed-point numerical type.

[0159] In one possible design, the second generation module 504 is specifically used for:

[0160] The third quantization factor is used to determine the quantization factor of the dequantization process performed by the dequantization function included in the activation function of the second RNN model.

[0161] The quantization factor is determined based on the fourth quantization factor, which is the quantization factor of the quantization function included in the activation function of the second RNN model.

[0162] Based on the above embodiments, this application also provides an electronic device for implementing the method for generating RNN models in the above figures, such as... Figure 6 The diagram shown is a structural schematic of another electronic device provided in an embodiment of this application. The electronic device 600 may include:

[0163] At least one processor 601; and a communication interface 603 communicatively connected to the at least one processor 601;

[0164] The at least one processor 601 causes the electronic device 600 to perform operations by executing instructions stored in the memory 602. Figures 3-4b The method shown.

[0165] In one possible design, the memory 602 is located outside the electronic device 600.

[0166] In one possible design, the electronic device 600 includes a memory 602 connected to the at least one processor 601, the memory 602 storing instructions executable by the at least one processor 601. (Appendix) Figure 6 The dashed line indicates that memory 602 is optional for electronic device 600.

[0167] The processor 601 and the memory 602 can be coupled through an interface circuit or integrated together; no restriction is imposed here.

[0168] It should be noted that, in this embodiment, the specific connection medium between the processor 601, memory 602, and communication interface 603 is not limited. This embodiment... Figure 6 The processor 601, memory 602, and communication interface 603 are connected via a bus 604. Figure 6 The connections between other components are shown in bold and are for illustrative purposes only, not as limiting information. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0169] It should be noted that, in the embodiments of this application, the processor mentioned can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0170] For example, the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0171] It should be noted that, in the embodiments of this application, the memory mentioned can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0172] It should be noted that, in the embodiments of this application, when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated into the processor.

[0173] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above description... Figure 6 The specific working process of the electronic device 600 after implementing the embodiments of the present invention can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. A more detailed description of the structure and function of the electronic device 600 can be found in the foregoing... Figure 2 Detailed description of the illustrated embodiments.

[0175] Based on the above embodiments, this application also provides a computer storage medium, including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the above... Figures 3-4bThe various method embodiments shown.

[0176] Based on the above embodiments, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the above... Figures 3-4b The various method embodiments shown.

[0177] Based on the above embodiments, this application also provides a chip, which is coupled to the memory of an electronic device and is used to read and execute program instructions stored in the memory, causing the electronic device to perform the above... Figures 3-4b The various method embodiments shown.

[0178] It should be understood that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0179] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of generating an RNN model, characterized by, include: A first recurrent neural network (RNN) model is generated; wherein the first RNN model is used for automatic speech recognition and / or semantic understanding, the first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function, any input matrix and any weight matrix of the first RNN model are quantized and then input into any of the matrix multiplication functions, the calculation result of any of the matrix multiplication functions is dequantized and then input into the matrix addition function, the calculation result of the matrix addition function is quantized and then input into the activation function, the calculation result of the activation function is dequantized and then used as the output matrix, the quantization process is used to quantize a floating-point number into a fixed-point number, and the dequantization process is used to dequantize a fixed-point number back into a floating-point number. Input the floating-point type of the input matrix into the first RNN model to train the model, and obtain the accuracy loss of the first RNN model during model training. If the accuracy loss is not less than a preset threshold, the weight matrix of the first RNN model is updated based on the accuracy loss to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold. A second RNN model is generated, and automatic speech recognition and / or semantic understanding are performed based on the second RNN model; wherein the second RNN model is the updated first RNN model after removing quantization and dequantization processing.

2. The method of claim 1, wherein, The step of updating the weight matrix of the first RNN model based on the accuracy loss to obtain the updated first RNN model includes: Obtain the gradient corresponding to any weight in the weight matrix of the first RNN model and the accuracy loss; wherein the gradient is the derivative of the accuracy loss with respect to any weight; The updated weight is obtained by updating any weight based on the gradient; wherein the updated weight is the sum of the gradient and any weight before the update.

3. The method of claim 1 or 2, wherein, The activation function includes an inverse quantization function, a linear activation function, and a quantization function; The calculation formula for the activation function is as follows: wherein, is a dequantization function, is a linear activation function, is a quantization function, is a calculation result of the matrix addition function after quantization processing.

4. The method of claim 3, wherein, The generation of the second RNN model includes: Obtain a first quantization factor and a second quantization factor when training the updated first RNN model; wherein, the first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function, and the second quantization factor is the quantization factor of the dequantization process of the calculation result of any matrix multiplication function before inputting the matrix addition function. The formula for calculating the matrix multiplication function of the second RNN model is determined based on the first quantization factor and the second quantization factor. Obtain the third quantization factor and the fourth quantization factor when training the updated first RNN model; wherein, the third quantization factor is the quantization factor of the calculation result of the matrix addition function before it is input into the activation function, and the fourth quantization factor is the quantization factor of the calculation result of the activation function before it is used as the output matrix. The quantization factor of the activation function of the second RNN model is determined based on the third quantization factor and the fourth quantization factor. Remove all quantization and dequantization processes from the updated first RNN model, except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, to generate the second RNN model.

5. The method of claim 4, wherein, The formula for calculating the matrix multiplication function of the second RNN model based on the first quantization factor and the second quantization factor includes: The formula for calculating the matrix multiplication function of the second RNN model is as follows: wherein, is a scaling factor for a weight matrix in the first quantization factor, is a zero point for a weight matrix in the first quantization factor, is a scaling factor for an input matrix in the first quantization factor, is a zero point for an input matrix in the first quantization factor ， is a scaling factor for a matrix multiplication result in the second quantization factor, is a zero point for a matrix multiplication result in the second quantization factor, is a weight matrix of a numerical type of fixed-point type, is an input matrix of a numerical type of fixed-point type, is a matrix multiplication result of a numerical type of fixed-point type.

6. The method as described in claim 4 or 5, characterized in that, The step of determining the quantization factor of the activation function of the second RNN model based on the third quantization factor and the fourth quantization factor includes: The third quantization factor is used to determine the quantization factor of the dequantization process performed by the dequantization function included in the activation function of the second RNN model. The quantization factor is determined based on the fourth quantization factor, which is the quantization factor of the quantization function included in the activation function of the second RNN model.

7. An apparatus for generating RNN models, characterized in that, include: A first generation module is used to generate a first recurrent neural network (RNN) model. The first RNN model is used for automatic speech recognition and / or semantic understanding. The first RNN model includes a matrix multiplication function, a matrix addition function, and an activation function. Any input matrix and any weight matrix are quantized and then input into any of the matrix multiplication functions. The calculation result of any of the matrix multiplication functions is dequantized and then input into the matrix addition function. The calculation result of the matrix addition function is quantized and then input into the activation function. The calculation result of the activation function is dequantized and then used as the output matrix. The quantization process is used to quantize floating-point numbers into fixed-point numbers, and the dequantization process is used to dequantize fixed-point numbers back into floating-point numbers. The training module is used to input the input matrix of floating-point type into the first RNN model for model training and to obtain the accuracy loss of the first RNN model during model training. An update module is used to update the weight matrix of the first RNN model based on the accuracy loss if the accuracy loss is not less than a preset threshold, so as to obtain an updated first RNN model, until the accuracy loss of the updated first RNN model during model training is less than the preset threshold. The second generation module is used to generate a second RNN model and perform automatic speech recognition and / or semantic understanding based on the second RNN model; wherein the second RNN model is the updated first RNN model after removing quantization and dequantization processing.

8. The apparatus as claimed in claim 7, characterized in that, The updated model is specifically used for: Obtain the gradient corresponding to any weight in the weight matrix of the first RNN model and the accuracy loss; wherein the gradient is the derivative of the accuracy loss with respect to any weight; The updated weight is obtained by updating any weight based on the gradient; wherein the updated weight is the sum of the gradient and any weight before the update.

9. The apparatus as claimed in claim 7 or 8, characterized in that, The activation function includes an inverse quantization function, a linear activation function, and a quantization function; The calculation formula for the activation function is as follows: in, It is an inverse quantization function. It is a linear activation function. For quantization function, This is the calculation result of the matrix addition function after quantization.

10. The apparatus as claimed in claim 9, characterized in that, The updated model is specifically used for: Obtain a first quantization factor and a second quantization factor when training the updated first RNN model; wherein, the first quantization factor is the quantization factor of any input matrix and any weight matrix of the updated first RNN model before inputting any matrix multiplication function, and the second quantization factor is the quantization factor of the dequantization process of the calculation result of any matrix multiplication function before inputting the matrix addition function. The formula for calculating the matrix multiplication function of the second RNN model is determined based on the first quantization factor and the second quantization factor. Obtain the third quantization factor and the fourth quantization factor when training the updated first RNN model; wherein, the third quantization factor is the quantization factor of the calculation result of the matrix addition function before it is input into the activation function, and the fourth quantization factor is the quantization factor of the calculation result of the activation function before it is used as the output matrix. The quantization factor of the activation function of the second RNN model is determined based on the third quantization factor and the fourth quantization factor. Remove all quantization and dequantization processes from the updated first RNN model, except for the quantization process performed by the quantization function included in the activation function and the dequantization process performed by the dequantization function, to generate the second RNN model.

11. The apparatus as claimed in claim 10, characterized in that, The updated model is specifically used for: The formula for calculating the matrix multiplication function of the second RNN model is as follows: in, This is the scaling factor for the weight matrix in the first quantization factor. The zeros of the weight matrix in the first quantization factor. is the scaling factor of the input matrix in the first quantization factor. The zeros of the input matrix in the first quantization factor ， This is the scaling factor for the matrix multiplication result in the second quantization factor. The zeros of the matrix multiplication result in the second quantization factor. The weight matrix is ​​a fixed-point numerical type. The input matrix is ​​a fixed-point numerical type. This is the result of a matrix multiplication with a fixed-point numerical type.

12. The apparatus as claimed in claim 10 or 11, characterized in that, The updated model is specifically used for: The third quantization factor is used to determine the quantization factor of the dequantization process performed by the dequantization function included in the activation function of the second RNN model. The quantization factor is determined based on the fourth quantization factor, which is the quantization factor of the quantization function included in the activation function of the second RNN model.

13. An electronic device, characterized in that, The electronic device includes at least one processor and a memory; The memory stores one or more computer programs; When one or more computer programs stored in the memory are executed by the at least one processor, the electronic device performs the method as described in any one of claims 1-6.

14. A computer storage medium, characterized in that, The computer storage medium includes computer instructions; when the computer instructions are executed on the computer, the computer causes the computer to perform the method as described in any one of claims 1-6.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-6.

Images