Neural network model end-side reasoning method and device, electronic equipment and storage medium

Abstract

The invention discloses a neural network model end-side reasoning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining initial data of an operator for reasoning, and determining a scaling factor according to the initial data; according to the scaling factor, scaling processing is carried out on the weight data subjected to convolution, and processed weight data is obtained; performing convolution according to the initial data and the processed weight data to obtain a convolution result; and scaling the convolution result according to the scaling factor to obtain an inference result of the operator based on the initial data. By the adoption of the method, the weight data are processed through the scaling factor in the initial stage, then the convolution result is subjected to reverse processing after convolution processing, the final reasoning result is obtained, the effective precision of reasoning is improved, and reasoning is more accurate.

Description

Technical Field

[0001] The present application relates to the technical field of neural network model processing, and in particular to a neural network model terminal-side reasoning method, device, electronic device and storage medium. Background Art

[0002] At present, the system-on-chip (SOC) on the terminal side, including the central processing unit (CPU), image processing unit (GPU) and neural-network processing unit (NPU), all support FP16 (Float16 precision, floating point single precision) calculation, which has obvious performance improvement and memory space release compared with FP32 (Float32 precision, floating point single precision).

[0003] However, there are also certain problems with using FP16 inference. Since the number of effective bits and dynamic range of FP16 are much smaller than those of FP32, the effective precision error of FP16 will increase, resulting in overflow and underflow problems when using FP16 inference for inference calculations, which in turn affects the use effect of the neural network model.

[0004] Therefore, how to effectively improve the FP16 reasoning accuracy is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a neural network model end-side reasoning method, device, electronic device and storage medium to solve the technical problem of low FP16 reasoning accuracy in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for inferring a neural network model on a terminal side, including:

[0007] Obtaining initial data for the operator to perform inference, and determining a scaling factor based on the initial data;

[0008] According to the scaling factor, scaling the weight data subjected to convolution is performed to obtain processed weight data;

[0009] Performing convolution according to the initial data and the processed weight data to obtain a convolution result;

[0010] The convolution result is scaled according to the scaling factor to obtain an inference result of the operator based on the initial data.

[0011] In a second aspect, an embodiment of the present application provides a neural network model terminal-side reasoning device, including:

[0012] A scaling determination module, used to obtain initial data for the operator to perform reasoning, and determine a scaling factor based on the initial data;

[0013] A first scaling module, configured to scale the convolved weight data according to the scaling factor to obtain processed weight data;

[0014] A convolution calculation module, used to perform convolution according to the initial data and the processed weight data to obtain a convolution result;

[0015] The second scaling module is used to scale the convolution result according to the scaling factor to obtain the inference result of the operator based on the initial data.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in any one of the above-mentioned neural network model end-side inference methods when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned neural network model end-side inference methods.

[0018] The embodiment of the present application provides a method, device, electronic device and storage medium for end-side reasoning of a neural network model. When reasoning is performed on the end-side of the neural network, the operator determines the scaling factor for scaling the weight data in the reasoning process based on the high-precision initial data, and then in the initial stage of the convolution reasoning calculation, the weight data is scaled using the scaling factor, and then the scaled weight data is used for convolution, and finally the convolution result is de-processed using the scaling factor to obtain the output reasoning result. Using the embodiment of the present application, the weight data is processed using the scaling factor in the initial stage, and then the convolution result is de-processed after the convolution, which improves the effective accuracy of the reasoning and makes the reasoning more accurate. At the same time, the introduction of the scaling factor in the initial stage avoids the impact on the performance, so that when reasoning based on FP16, the performance is guaranteed while the precision and accuracy of the reasoning are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a method for inferring a neural network model on the terminal side provided in an embodiment of the present application;

[0020] Figure 2 This is a schematic diagram of the form of FP32 format data provided in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the form of FP32 format data provided in an embodiment of the present application;

[0022] Figure 4 is a flowchart of the steps of determining a scaling factor provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the convolution calculation process provided by the embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of a scenario in which two operators perform reasoning according to an embodiment of the present application;

[0025] Figure 7 It is a structural diagram of a neural network model terminal-side reasoning device provided in an embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] Fig. 9 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] In the related art, when performing convolution or quasi-convolution reasoning on the end side of a neural network (such as a CNN network (Convolutional Neural Network), LSTM network (Long short-term memory), Transformer network, etc.), using FP16 can effectively improve performance and reduce memory usage. However, there is a certain error in the effective accuracy of reasoning, which results in inaccurate reasoning caused by errors when the neural network model is used.

[0032] Therefore, how to simply and effectively improve the FP16 reasoning accuracy on the neural network side is an urgent problem that needs to be solved.

[0033] In order to solve the technical problems existing in the related art, the present application embodiment provides a neural network model terminal side reasoning method, see Figure 1 , Figure 1 1 is a flow chart of a method for inferring a neural network model on the terminal side provided in an embodiment of the present application, the method comprising steps 101 to 104;

[0034] Step 101, obtaining initial data for the operator to perform reasoning, and determining a scaling factor based on the initial data.

[0035] Among them, the deep learning algorithm is composed of computing units, which we call operators (OP for short). In the network model, the operator corresponds to the computing logic in the layer, for example: the convolution layer (ConvolutionLayer) is an operator; the weight summation process in the fully-connected layer (FC layer) is also an operator.

[0036] The SOC on the end side of the neural network, including CPU, GPU and NPU, all support FP16 and FP32 calculations. After considering the advantages and disadvantages of the two, FP16 is used in actual use to implement end-side reasoning because it has better advantages in performance and memory. However, FP16 has a loss in precision compared to FP32. Therefore, when performing reasoning calculations, FP16 may cause reasoning errors due to low precision. Therefore, it is necessary to reasonably optimize the FP16 reasoning process to reduce errors or errors caused by low precision.

[0037] Therefore, the present application adds a scaling factor, such as an amplification factor, to the inference convolution process to make an equivalent change, thereby avoiding overflow during the convolution process. At the same time, when performing the scaling process, by applying the scaling factor to the weight data of the convolution calculation in the initial stage, such as multiplying the amplification factor with the weight data, rather than the input initial data itself, the performance of the neural network model will not be affected. Then, the convolution process is performed after the scaling process, and finally the result of the convolution process is processed here with the scaling factor, such as dividing by the amplification factor to obtain the final inference result. Compared with the multiplication and addition process in the convolution process, the amount of calculation of the division operation performed at this time is relatively small, so that the impact on performance is also small. Compared with conventional FP16 reasoning, while ensuring performance and memory usage, the effective accuracy of reasoning is improved, making the reasoning more accurate.

[0038] In one embodiment, the reasoning accuracy during FP16 reasoning is improved by processing. At this time, the initial data for reasoning calculation is first obtained, and the scaling factor for processing is determined based on the obtained initial data. Due to the characteristics of the neural network model, underflow or overflow may occur during reasoning calculation, and the occurrence of underflow and overflow is caused by the difference or inaccuracy of precision.

[0039] For example, for FP16 and FP32, the difference in data format between the two leads to different data accuracy. Figure 2 As shown, the FP16 format is Figure 3 As shown. Based on the difference between the two formats, the absolute minimum value of FP16 is 2, calculated according to the number of exponents and mantissas in the format. -34 =5.960464e -8 , the absolute minimum for FP32 is 2 -149 =1.401298e -45 . Therefore, for FP16, it falls within (1.401298e -45 , 5.960464e -8 ) cannot be represented, that is, all numbers are displayed as 0, which is underflow. When reasoning based on FP16, the accuracy of data reasoning will be affected due to underflow for data that falls in the interval that cannot be displayed. Therefore, it is necessary to process it reasonably to avoid underflow and improve the reasoning accuracy and effect.

[0040] In the actual convolution processing, the convolution calculation process can be shown as follows:

[0041]

[0042] Among them, O00 is the convolution result, is the initial data, is the weight data.

[0043] The initial data is The weight data is When , the process and result of FP32 convolution calculation are:

[0044]

[0045] Due to the difference in data precision, the process and result of FP16 convolution calculation are as follows:

[0046]

[0047] Obviously, the result obtained based on FP32 convolution calculation is correct, while the result obtained based on FP16 convolution calculation is wrong. Therefore, when using FP16 for convolution calculation, it is necessary to avoid the above situation to ensure the accuracy of neural network model processing.

[0048] Specifically, taking the improvement of the FP16 reasoning accuracy on the end side of the neural network model as an example, when processing, the initial data for reasoning is first obtained, and the obtained initial data is data in FP32 format, and then the scaling factor is determined based on the initial data. In practical applications, the scaling factor can be an enlargement factor or a reduction factor. For the sake of convenience, the scaling factor is taken as an example for explanation.

[0049] In the actual processing process, the scaling factor is obtained based on the initial data. While considering the initial data, the weight data used in the convolution processing will also be used as the basis for determining the scaling factor. The specific determination process of the scaling factor can be as follows: Figure 4 As shown, the process includes steps 401 to 402.

[0050] Step 401, obtain initial data for inference, and convert the format of the initial data to obtain data after format conversion, wherein the data after format conversion is second format data, the initial data is second format data, and the data accuracy of the first format data is lower than the data accuracy of the second format data.

[0051] Specifically, when the initial data for reasoning is obtained, the data format is converted to obtain the format-converted data, and the format-converted data is first format data, and the initial data obtained in advance is second format data, wherein the data accuracy of the first format data is lower than the data accuracy of the second format data, such as the initial data is FP32 format data, and the first format data obtained after the format conversion is FP16 format data.

[0052] It is noted that the initial data for convolution processing can be data in a conventional format, such as decimal data, and then the data in FP32 format and FP16 format are obtained through format conversion, and then the scaling factor for subsequent scaling processing is determined by obtaining relevant information in the data in different formats.

[0053] Step 402, obtaining the index value in the initial data, and determining the scaling factor according to the weight data and the index value.

[0054] Specifically, when determining the scaling factor, it is determined based on the set method, and then the required related data is obtained for calculation and processing, wherein the formula for determining the scaling factor can be as follows:

[0055] log 2 factor=123-exponent(average(abs(filter)));

[0056]

[0057] Among them, exponent is the 8-bit exponent in FP32 format, average is the average, abs is the absolute value, filter is the weight data, and facter is the scaling factor.

[0058] It should be noted that due to the difference in data accuracy between FP16 and FP32, there are certain differences in the display of weight data when reasoning is performed based on the two methods, but they are roughly the same. Therefore, for the sake of ease of description, the weight data under FP16 and FP32 are described here as the same data.

[0059] When determining the scaling factor based on the above formula, the weight data and exponent value of the initial data in FP32 format are obtained, and the calculation is performed according to the above formula to determine the current corresponding scaling factor. Wherein, when performing the calculation to determine the scaling factor, it includes: performing the calculation according to the weight data and the exponent value to obtain the calculation result; determining whether to perform scaling processing on the weight data according to the calculation result, and when it is determined to perform scaling processing, setting the calculation result as the scaling factor.

[0060] Specifically, the scaling factor is determined based on the above formula. At the same time, not all situations require scaling. Therefore, in the process of determining the scaling factor, it is necessary to determine whether scaling is required based on actual conditions. When determining whether scaling is required, it is determined based on the result obtained by calculation. First, the first result log is obtained by calculation. 2 factor, and then log the first result 2factor is compared with a preset threshold value to determine whether scaling is required, and then when it is determined that scaling is required, the scaling factor is determined.

[0061] Exemplarily, when determining whether scaling is required, it includes: comparing the calculation result with a preset threshold, and determining whether to scale the weight data based on the comparison result; if the calculation result is less than or equal to the preset threshold, determining not to scale the weight data; if the calculation result is greater than the preset threshold, determining to scale the weight data.

[0062] Among them, the preset threshold is set to 0, and whether scaling is required is determined by determining whether the calculation result is greater than 0. When it is determined that scaling is required, calculation is performed based on the above formula to determine the scaling factor. When it is determined that scaling is not required, convolution processing is performed directly based on the FP16 format data of the initial data to obtain the current convolution inference result.

[0063] Step 102: scaling the convolved weight data according to the scaling factor to obtain processed weight data.

[0064] In one embodiment, after the scaling factor is determined based on the method described in step 101, the weight data to be convolved is scaled according to the scaling factor to obtain the weight data after the scaling process is completed.

[0065] From the above description, it can be seen that there are differences in the weight data in FP32 format and FP16 format, but the specific difference in the data is small, which is specifically reflected in the data accuracy. Therefore, the weight data in the two formats are set to be the same. Of course, they can be distinguished in the actual processing process.

[0066] Specifically, in order to avoid affecting the performance during the scaling process, the weight data is scaled, which is an operation performed during the initialization phase to avoid affecting the inference performance. At this time, after obtaining the scaling factor, when performing the scaling process, the scaling factor is multiplied by the weight data. When the scaling factor is e2, the processed scaling factor is:

[0067]

[0068] Step 103, performing convolution based on the initial data and the processed weight data to obtain a convolution result.

[0069] In one embodiment, after the scaling factor is determined and the scaling process of the weight data is completed according to the scaling factor, the convolution process is performed on the processed initial data of the weight data to obtain the convolution result after the scaling factor is added. When performing the convolution process, the initial data after the format conversion is convolved with the processed scaling factor to obtain the corresponding convolution result.

[0070] Specifically, when performing convolution processing, the first format data of the initial data is convolved with the scaled weight data. Exemplarily, the initial data for convolution at this time is FP16 format data, and the current convolution result is obtained based on the convolution calculation method.

[0071] Taking the above example, the convolution result obtained at this time is:

[0072]

[0073] That is, when performing the convolution process, it includes: performing convolution process on the format-converted data and the processed weight data to obtain the convolution result. That is, performing convolution process on the format-converted data and the scaled weight data to obtain the convolution result.

[0074] Step 104, scaling the convolution result according to the scaling factor to obtain the inference result of the operator based on the initial data.

[0075] In one embodiment, after performing convolution calculation to obtain a corresponding convolution result, the convolution result is scaled according to a scaling factor to obtain an inference result of the initial data.

[0076] Exemplarily, when the scaling factor is an amplification factor, when processing steps 101 to 103, the weight data is multiplied by the scaling factor so that the convolution result obtained during the convolution processing is in the data display interval corresponding to the FP16 format, that is, to avoid underflow. Then, when the final output is performed, the convolution result is divided by the scaling factor, so that the corresponding inference result can be accurately obtained, avoiding the occurrence of inaccurate inference due to underflow.

[0077] After completing the above convolution calculation, when the scaling factor is removed, the convolution result is divided by the scaling factor. The specific inference result at this time is:

[0078]

[0079] That is, when the convolution inference calculation is performed based on the method described in step 101 to step 104, the convolution calculation process at this time is:

[0080]

[0081] At the same time, the specific convolution calculation process can be as follows Figure 5 As shown, specifically, the convolution calculation process at this time is: first obtain the relevant data in the FP32 format of the initial data, determine the scaling factor used at this time by calculation, and then scale the weight data in the FP16 format based on the scaling factor, and then perform convolution calculation after completing the scaling process, and finally scale the result data obtained by the convolution calculation here to output the current reasoning result. Among them, when the scaling factor is the amplification factor, the weight data is multiplied by the scaling factor to achieve data amplification, and then divided by the scaling factor after completing the convolution calculation to obtain the final reasoning result.

[0082] Exemplarily, the weights in FP32 format in the initial data are obtained (Filter(fp32)), and then calculated in the set manner to obtain the scaling factor (factor), and then the scaling factor is multiplied by the weight data to obtain the processed weight data (convert-fp16) used for convolutional inference, and then in the inference layer (inference), the format-converted data (Input(fp16)) is convolved with the weight data (convert-fp16) to obtain the corresponding convolution result, and the scaling factor is removed from the convolution result to obtain the final output result (Output(fp16)).

[0083] Furthermore, in one embodiment, after obtaining the convolution result, in order to ensure the performance of the neural network model, other methods can be used to process the SOC that cannot adapt well to the division operation to ensure that the performance is not affected. Specifically, after obtaining the convolution result, the convolution result can be processed by subtraction and shifting to obtain the final inference result. Specifically including: obtaining the first exponent bit of the convolution result, and obtaining the second exponent bit of the scaling factor; performing subtraction and shifting processing based on the first exponent bit and the second exponent bit to obtain the third exponent bit; replacing the first exponent bit based on the third exponent bit to obtain the inference result based on the initial data.

[0084] Taking the above example as an example, when the scaling factor (magnification factor) is e 2 When the convolution result obtained by magnification and convolution is 1.96e -5 At this time, if the processing is performed based on the method described in step 104, by -5 Divide by e 2 , and obtain the final inference result.

[0085] At this time, when processing, the convolution result is determined to be 1.96e -5The first exponent of is -5, and the second exponent of the scaling factor is 2. Based on the characteristics of determining the scaling factor, the exponent of the inference result can be determined to be (-5)-2=(-7). Then the inference result obtained at this time is 1.96e -7 , that is, the first exponent bit in the convolution result is replaced by the third exponent bit as the final inference result.

[0086] It should be noted that in the above embodiments, a method of improving the FP16 reasoning accuracy on the end side of the neural network model is described, and this method is also applicable to improving FP8 and other related convolutional reasoning processes for improving reasoning accuracy, and the specific process is similar to the above process and is not described in detail here.

[0087] In addition, in the above-described embodiments, the convolution process within an operator is described. However, in practical applications, this method is also applicable to the processing between operators, such as Figure 6 The scene shown in Figure 6 In the described situation, for FP32, the calculation result of operator 1 is very small, and operator 2 outputs a relatively large value. If FP16 is used for reasoning and calculation at this time, the output result of operator 1 will underflow, that is, the output value is 0, which will cause the input of operator 2 to be 0, that is, the reasoning result will be abnormal.

[0088] Therefore, at this time, when operator 1 and operator 2 meet the fusion conditions, the two operators can be fused, that is, an amplification factor is added during the calculation process of operator 1 so that operator 2 can get the correct input, and then in the last step before operator 2 outputs the inference result, this amplification factor is removed to obtain the correct output result.

[0089] Specifically, when performing fusion reasoning between operators, it includes: when determining that the first operator is to be scaled, determining the scaling factor based on the first operator, and controlling the first operator to perform convolution processing based on the scaling factor to obtain a first result; inputting the first result into the second operator for convolution processing to obtain a second result, wherein the second operator is the next operator connected to the first operator, and the output of the second operator is the reasoning result of the current neural network model; scaling the second result according to the scaling factor to obtain the reasoning result output by the second operator.

[0090] Exemplarily, when it is determined that the first operator and the second operator can be fused, the scaling factor is determined according to the relevant data in the first operator, wherein the process of determining the scaling factor is the same as the process described in step 101, and when performing reasoning, the first operator outputs a result based on the scaling factor to obtain a first result, and the first result obtained is similar to the convolution result described in step 103, that is, the output result of the first operator is a result scaled based on the scaling factor, such as amplified based on the magnification factor, and then the first result is used as the input of the second operator to perform convolution processing in the second operator, and when the second operator finally outputs, the second result obtained by convolution is scaled based on the scaling factor to obtain the final output of the second factor, that is, the inference result of the second factor.

[0091] In summary, the present application discloses a method for end-side reasoning of a neural network model. When reasoning is performed on the end-side of the neural network, a scaling factor for scaling the weight data in the reasoning process is determined based on high-precision initial data, and then in the initial stage of the convolution reasoning calculation, the weight data is scaled using the scaling factor, and then the scaled weight data is used for convolution processing, and finally the convolution result is de-processed using the scaling factor to obtain the output reasoning result. Using the embodiment of the present application, the weight data is processed using the scaling factor in the initial stage, and then the convolution result is de-processed after the convolution processing, which improves the effective accuracy of the reasoning and makes the reasoning more accurate. At the same time, the introduction of the scaling factor in the initial stage avoids the impact on the performance, so that when reasoning based on FP16, the performance is guaranteed while improving the precision and accuracy of the reasoning.

[0092] According to the method described in the above embodiment, this embodiment will be further described from the perspective of the neural network model end-side reasoning device. The neural network model end-side reasoning device can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0093] See also Figure 7 , Figure 7 is a structural diagram of a neural network model terminal-side reasoning device provided in an embodiment of the present application, such as Figure 7 As shown, the neural network model terminal-side reasoning device 700 provided in the embodiment of the present application includes:

[0094] A scaling determination module 701 is used to obtain initial data for the operator to perform reasoning and determine a scaling factor based on the initial data;

[0095] A first scaling module 702, configured to scale the convolved weight data according to a scaling factor to obtain processed weight data;

[0096] A convolution calculation module 703, used to perform convolution according to the initial data and the processed weight data to obtain a convolution result;

[0097] The second scaling module 704 is used to scale the convolution result according to the scaling factor to obtain the inference result of the operator based on the initial data.

[0098] In one embodiment, the scaling determination module 701 is further configured to:

[0099] Acquire initial data for inference, and perform format conversion on the initial data to obtain format-converted data, wherein the converted data is first-format data, the initial data is second-format data, and data accuracy of the first-format data is lower than data accuracy of the second-format data;

[0100] Get the exponent value in the initial data and determine the scaling factor based on the weight data and the exponent value.

[0101] In one embodiment, the scaling determination module 701 is further configured to:

[0102] Calculate according to the weight data and the index value to obtain the calculation result;

[0103] Whether to perform scaling processing on the weight data is determined according to the calculation result, and when it is determined to perform scaling processing, the calculation result is set as the scaling factor.

[0104] In one embodiment, the scaling determination module 701 is further configured to:

[0105] Compare the calculation result with the preset threshold, and determine whether to perform scaling processing on the weight data according to the comparison result;

[0106] If the calculated result is less than or equal to the preset threshold, it is determined that the weight data is not scaled;

[0107] If the calculated result is greater than a preset threshold, it is determined to perform scaling processing on the weight data.

[0108] In one embodiment, the convolution calculation module 703 is further used for:

[0109] The format-converted data is convolved with the processed weight data to obtain the convolution result.

[0110] In one embodiment, the second scaling module 704 is further configured to:

[0111] Obtain the first exponent bit of the convolution result, and obtain the second exponent bit of the scaling factor;

[0112] Perform subtraction and shift processing according to the first exponent bit and the second exponent bit to obtain a third exponent bit;

[0113] The first exponent bit is replaced based on the third exponent bit to obtain the inference result of the operator based on the initial data.

[0114] In one embodiment, the neural network model terminal-side reasoning device 700 is also used for:

[0115] When it is determined that the first operator performs scaling processing, a scaling factor is determined based on the first operator, and the first operator is controlled to perform convolution processing based on the scaling factor to obtain a first result;

[0116] Inputting the first result into the second operator for convolution processing to obtain a second result, wherein the second operator is the next operator connected to the first operator, and the output of the second operator is the inference result of the current neural network model;

[0117] The second result is scaled according to the scaling factor to obtain an inference result output by the second operator.

[0118] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0119] Also, see Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802. The processor 801 is electrically connected to the memory 802.

[0120] The processor 801 is the control center of the electronic device 800. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 800 and processes data by running or loading applications stored in the memory 802 and calling data stored in the memory 802, thereby monitoring the electronic device 800 as a whole.

[0121] In this embodiment, the processor 801 in the electronic device 800 will load the instructions corresponding to the processes of one or more applications into the memory 802 according to the following steps, and the processor 801 will run the application stored in the memory 802, thereby implementing any step in the neural network model end-side inference method provided in the above embodiment.

[0122] The electronic device 800 can implement the steps in any embodiment of the neural network model end-side inference method provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any neural network model end-side inference method provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0123] See also Fig. 9 , Fig. 9 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, Fig. 9 The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the neural network model terminal side reasoning method provided in the above embodiment. The electronic device 900 can be a mobile terminal such as a smart phone or a laptop computer.

[0124] The RF circuit 910 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 910 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 910 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0125] The memory 920 can be used to store software programs and modules, such as program instructions / modules corresponding to the end-side reasoning method of the neural network model in the above-mentioned embodiment. The processor 980 executes various functional applications and performs end-side reasoning of the neural network model by running the software programs and modules stored in the memory 920.

[0126] The memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 920 may further include a memory remotely arranged relative to the processor 980, and these remote memories may be connected to the electronic device 900 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The input unit 930 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 931 and other input devices 932. The touch-sensitive surface 931, also known as a touch display screen or a touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 931), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 931 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 980, and can receive and execute commands sent by the processor 980. In addition, the touch-sensitive surface 931 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 931, the input unit 930 may further include other input devices 932. Specifically, the other input devices 932 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0128] The display unit 940 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 900, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 940 may include a display panel 941, and optionally, the display panel 941 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 931 may cover the display panel 941, and when the touch-sensitive surface 931 detects a touch operation on or near it, it is transmitted to the processor 980 to determine the type of the touch event, and then the processor 980 provides corresponding visual output on the display panel 941 according to the type of the touch event. Although in the figure, the touch-sensitive surface 931 and the display panel 941 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 931 and the display panel 941 can be integrated to implement input and output functions.

[0129] The electronic device 900 may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 941 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 900, they will not be repeated here.

[0130] The audio circuit 960, the speaker 961, and the microphone 962 can provide an audio interface between the user and the electronic device 900. The audio circuit 960 can transmit the electrical signal converted from the received audio data to the speaker 961, which is converted into a sound signal for output; on the other hand, the microphone 962 converts the collected sound signal into an electrical signal, which is received by the audio circuit 960 and converted into audio data, and then the audio data is processed by the output processor 980 and sent to another terminal through the RF circuit 99, or the audio data is output to the memory 920 for further processing. The audio circuit 960 may also include an earplug jack to provide communication between an external headset and the electronic device 900.

[0131] The electronic device 900 can help users receive requests, send information, etc. through the transmission module 970 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Although the transmission module 970 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 900 and can be omitted as needed without changing the essence of the invention.

[0132] The processor 980 is the control center of the electronic device 900. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 920, and calling data stored in the memory 920, it executes various functions of the electronic device 900 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 980 may include one or more processing cores; in some embodiments, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 980.

[0133] The electronic device 900 also includes a power supply 990 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 980 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 990 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0134] Although not shown, the electronic device 900 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, one or more of which are stored in the memory, and are configured to be executed by one or more processors to implement any step in the end-side reasoning method of the neural network model provided in the above embodiment.

[0135] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0136] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which multiple instructions are stored, and the instructions can be executed by the processor to implement any step in the end-side reasoning method of the neural network model provided in the above embodiments.

[0137] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0138] Since the instructions stored in the storage medium can execute the steps in any embodiment of the neural network model end-side inference method provided in the embodiments of the present application, the beneficial effects that can be achieved by any neural network model end-side inference method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0139] The above is a detailed introduction to a neural network model end-side reasoning method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. In addition, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present application.

Claims

1. A method for inferencing a neural network model on the client side, characterized in that: include: Obtaining initial data for the operator to perform inference, and determining a scaling factor based on the initial data; According to the scaling factor, scaling the weight data subjected to convolution is performed to obtain processed weight data; Performing convolution according to the initial data and the processed weight data to obtain a convolution result; The convolution result is scaled according to the scaling factor to obtain an inference result of the operator based on the initial data.

2. The method according to claim 1, characterized in that Initial data for the operator to perform reasoning is obtained, and a scaling factor is determined based on the initial data, including: Acquire initial data for inference, and perform format conversion on the initial data to obtain format-converted data, wherein the format-converted data is first-format data, the initial data is second-format data, and data precision of the first-format data is lower than data precision of the second-format data; An exponent value in the initial data is obtained, and a scaling factor is determined according to the weight data and the exponent value.

3. The method according to claim 2, characterized in that The determining of the scaling factor according to the weight data and the exponent value comprises: Perform calculation according to the weight data and the index value to obtain a calculation result; Determine whether to perform scaling processing on the weight data according to the calculation result, and when it is determined to perform scaling processing, set the calculation result as a scaling factor.

4. The method according to claim 3, characterized in that The determining whether to perform scaling processing on the weight data according to the calculation result includes: Comparing the calculation result with a preset threshold, and determining whether to perform scaling processing on the weight data according to the comparison result; If the calculation result is less than or equal to the preset threshold, determining not to perform scaling processing on the weight data; If the calculation result is greater than the preset threshold, it is determined to perform scaling processing on the weight data.

5. The method according to claim 2, characterized in that The convolving the initial data with the processed weight data to obtain a convolution result includes: The format-converted data is convolved with the processed weight data to obtain the convolution result.

6. The method according to claim 1, characterized in that After convolving the initial data with the processed weight data to obtain the convolution result, the method further includes: Obtaining a first exponent bit of the convolution result, and obtaining a second exponent bit of the scaling factor; Perform subtraction and shift processing according to the first exponent bit and the second exponent bit to obtain a third exponent bit; The first exponent bit is replaced based on the third exponent bit to obtain an inference result of the operator based on the initial data.

7. The method according to claim 1, characterized in that The method further comprises: When it is determined that the first operator performs scaling processing, a scaling factor is determined based on the first operator, and the first operator is controlled to perform convolution processing based on the scaling factor to obtain a first result; Inputting the first result into the second operator for convolution processing to obtain a second result, wherein the second operator is the next operator connected to the first operator, and the output of the second operator is the inference result of the current neural network model; The second result is scaled according to the scaling factor to obtain an inference result output by the second operator.

8. A neural network model end-side reasoning device, characterized in that: include: A scaling determination module, used to obtain initial data for the operator to perform reasoning, and determine a scaling factor based on the initial data; A first scaling module, configured to scale the convolved weight data according to the scaling factor to obtain processed weight data; A convolution calculation module, used to perform convolution according to the initial data and the processed weight data to obtain a convolution result; The second scaling module is used to scale the convolution result according to the scaling factor to obtain the inference result of the operator based on the initial data.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images