Data processing device and data processing method

By designing a data processing device in a convolutional neural network, using M multipliers and M-1 adders for convolution processing, the resource waste problem caused by the size of the convolution kernel is less than 2n, and the resource utilization rate is improved and computational efficiency.

CN114730331BActive Publication Date: 2024-11-26HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980102503.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-18
Publication Date
2024-11-26
Estimated Expiration
2039-12-18

AI Technical Summary

Technical Problem

In convolutional neural networks, when the processor is used for convolutional operations, since the size of the convolution kernel is usually less than 2n, the multiplication unit and addition unit in the processor are partially idle, causing waste of resources and reducing resource utilization.

Method used

Design a data processing device, use M multipliers and M-1 adders for convolution processing, and calculate the convolution result through an additional adder, avoiding the use of unnecessary multipliers and adders in traditional devices. Suitable for processing of convolution kernels with size M+1 and matrices to be convolutionized.

Benefits of technology

It effectively avoids resource waste, improves resource utilization, and ensures the accuracy and efficiency of convolutional calculations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114730331B_ABST
    Figure CN114730331B_ABST
Patent Text Reader

Abstract

The present application provides a data processing device and a data processing method in the field of artificial intelligence. The present application provides a data processing device for performing convolution processing on a matrix to be convolved according to a two-dimensional convolution kernel, the two-dimensional convolution kernel includes a first parameter and M second parameters, the matrix to be convolved includes a first eigenvalue and M second eigenvalues, the M second parameters correspond to the M second eigenvalues ​​one-to-one, the first parameter corresponds to the first eigenvalue, the data processing device includes: M multipliers and M-1 first adders, for performing multiplication and accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain a multiplication and accumulation result; a second adder, for adding the multiplication and accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved. The data processing device and data processing method provided by the present application help to avoid resource waste and improve resource utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to data computing in the field of artificial intelligence, and more specifically, to a data processing device and a data processing method. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] As an important branch of artificial intelligence, neural network (NN) is a network structure that imitates the behavioral characteristics of animal neural networks to process information. The structure of a neural network is composed of a large number of nodes (or neurons) connected to each other. It processes information by learning and training input information based on a specific operation model. A neural network includes an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving input signals, the output layer is responsible for outputting the calculation results of the neural network, and the hidden layer is responsible for the calculation process such as learning and training. It is the memory unit of the network. The memory function of the hidden layer is represented by a weight matrix, and usually each neuron corresponds to a weight coefficient.

[0004] Among them, the convolutional neural network (CNN) is a multi-layer neural network, each layer consists of multiple two-dimensional planes, and each plane consists of multiple independent neurons. Each feature plane can be composed of some neural units arranged in a rectangular shape. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.

[0005] In a convolutional neural network, the processor usually performs a convolution operation by converting the convolution of feature information and weights in the feature plane into a matrix multiplication operation between a signal matrix and a weight matrix. In the specific matrix multiplication operation, the signal matrix and the weight matrix are processed in blocks to obtain multiple fractal signal matrices and fractal weight matrices, and then matrix multiplication and accumulation operations are performed on the multiple fractal signal matrices and fractal weight matrices.

[0006] At present, when the processor performs convolution on the input layer, each processing engine (PE) in the processor used for convolution operation usually includes 2 n Specifically, each PE of the processor used for convolution calculations includes 2 n Multiplication unit and 2 n -1 addition unit. However, the size of the convolution kernel used in convolution kernel operations is generally less than 2 n This causes some multiplication units and addition units to idle when PE performs convolution operations, resulting in a waste of processor resources.

[0007] For example, Fig.15 As shown, when the PE performing convolution operation includes 16 MACs and the convolution kernel size is 3*3, at least multiplication unit 9 to multiplication unit 15, and addition unit 8, addition unit 9, addition unit 10 and addition unit 12 in the PE will idle, resulting in a waste of processing resources. Summary of the invention

[0008] The present application provides a data processing device and a data processing method, which help to avoid waste of resources and improve resource utilization.

[0009] In a first aspect, the present application provides a data processing device, the data processing device is used to perform convolution processing on a matrix to be convolved according to a two-dimensional convolution kernel, the two-dimensional convolution kernel includes a first parameter and M second parameters, the matrix to be convolved includes a first eigenvalue and M second eigenvalues, the M second parameters correspond to the M second eigenvalues ​​one-to-one, and the first parameter corresponds to the first eigenvalue. The data processing device includes: M multipliers and M-1 first adders, which are used to perform multiplication and accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain a multiplication and accumulation result; a second adder, which is used to add the multiplication and accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved. Wherein, M is a positive integer greater than 1.

[0010] When the data processing device performs convolution processing on a matrix to be convolved of size M+1 according to a convolution kernel of size M+1, the convolution result of M parameters in the convolution kernel and M eigenvalues ​​in the matrix to be convolved is calculated by M multipliers and M-1 adders, and the sum of the convolution result and the M+1th eigenvalue in the matrix to be convolved is calculated by another adder, and finally the sum is used as the convolution result of the convolution kernel on the matrix to be convolved. This makes it possible to avoid using a plurality of traditional data processing devices including M+1 multipliers and M adders for calculation when the size of the convolution kernel that needs to be convolved is one more than the number of multiplication and accumulation units in the data processing device, thereby avoiding waste of resources and improving resource utilization.

[0011] Among them, the matrix to be convolved may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, and so on.

[0012] In some possible implementations, the first parameter is equal to 1. That is, the first parameter corresponding to the first eigenvalue input to the second adder is 1. In this way, the accuracy of the convolution result calculated by the data processing device can be guaranteed.

[0013] In some possible implementations, the device also includes a processor, and the processor is used to: determine the two-dimensional convolution kernel based on an initial convolution kernel, the initial convolution kernel includes a first initial parameter and M second initial parameters, the M second parameters correspond one-to-one to the M second initial parameters, the first parameter corresponds to the first initial parameter, the second parameter is equal to the quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter is not zero.

[0014] When the data processing device in this implementation is used to calculate the convolution result of the initial convolution kernel and the matrix to be convolved, all parameters in the initial convolution kernel can be divided by one of the non-zero parameters (i.e., the first initial parameter) through the processor, so that one parameter (i.e., the first parameter) in the obtained two-dimensional convolution kernel can be 1. In this way, when the above-mentioned M multipliers and M adders are used to calculate the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, a more accurate value can be obtained.

[0015] Optionally, the processor can also be used to: determine the convolution result of the initial convolution kernel and the matrix to be convolved based on the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, wherein the convolution result of the initial convolution kernel and the matrix to be convolved is equal to the product of the convolution result of the two-dimensional convolution kernel and the matrix to be convolved and the first initial parameter.

[0016] In some possible implementations, when the quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, before determining the two-dimensional convolution kernel according to the initial convolution kernel, the processor is further configured to: reduce the M second parameters or the M second initial parameters by m times, wherein the quotient of any second initial parameter reduced by m times and the first initial parameter is not greater than the first threshold. Wherein, m may be a positive integer.

[0017] The data processing device in this implementation can first reduce all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is greater than the first threshold, so that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than the first threshold. The first threshold can be less than or equal to the maximum expressible value of the processor, or the first threshold can be less than or equal to the maximum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the maximum expressible value range of the data processing device.

[0018] Alternatively, the data processing device in the implementation mode may reduce all second parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is greater than the first threshold value, so that the reduced second parameters are not greater than the first threshold value. The first threshold value may be less than or equal to the maximum expressible value of the processor, or the first threshold value may be less than or equal to the maximum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the maximum expressible value range of the data processing device.

[0019] In the case where the processor is also used to reduce the M second parameters or the M second initial parameters by m times, before adding the multiplication-accumulation result and the first eigenvalue, the processor is also used to: reduce the first eigenvalue by m times; correspondingly, the second adder is specifically used to add the multiplication-accumulation result and the reduced first eigenvalue. This is because when the second initial parameter or the second parameter is reduced by m times, the first parameter should also be reduced by m times accordingly, so the processing result of the first parameter on the first eigenvalue should also be reduced by m times. The data processing device of the present application directly reduces the first eigenvalue by m times to ensure the accuracy of the convolution result.

[0020] Optionally, the processor may reduce the first eigenvalue by a shift operation. Specifically, the first eigenvalue may be shifted left.

[0021] In some possible implementations, when the quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, before determining the two-dimensional convolution kernel according to the initial convolution kernel, the processor is further configured to: expand the M second parameters or the M second initial parameters by n times, wherein the quotient of any second initial parameter expanded by n times and the first initial parameter is not less than the second threshold. Wherein, n may be a positive integer.

[0022] The data processing device in this implementation can first expand all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is less than the second threshold value, so that the quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold value, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than the second first threshold value. The second threshold value can be greater than or equal to the minimum expressible value of the processor, or the second threshold value can be greater than or equal to the minimum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the minimum expressible value range of the data processing device.

[0023] Alternatively, the data processing device in this implementation may reduce all second parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is less than the second threshold value, so that the reduced second parameters are not less than the second threshold value. The second threshold value may be greater than or equal to the minimum expressible value of the processor, or in other words, the second threshold value may be greater than or equal to the minimum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the minimum expressible value range of the data processing device.

[0024] In some possible implementations, before adding the multiplication-accumulation result and the first eigenvalue, the processor is further used to: expand the first eigenvalue by n times; correspondingly, the second adder is specifically used to add the multiplication-accumulation result and the expanded first eigenvalue.

[0025] This is because when the second initial parameter or the second parameter is enlarged by m times, the first parameter should also be enlarged by m times accordingly, so the processing result of the first parameter on the first eigenvalue should also be enlarged by m times. The data processing device of the present application directly enlarges the first eigenvalue by m times to ensure the accuracy of the convolution result.

[0026] Optionally, the processor may perform an expansion process on the first eigenvalue by a shift operation. Specifically, the first eigenvalue may be shifted right.

[0027] In some possible implementations, the processor includes a combination of one or more of the following: a central processing unit, a graphics processing unit, or a neural network processor.

[0028] In some possible implementations, M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix. Optionally, the M multipliers and the M-1 first adders may constitute a product-accumulator.

[0029] In some possible implementations, M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix. Optionally, the M multipliers and the M-1 first adders constitute three multiplication accumulators, wherein each of the multiplication accumulators includes M / 3 multipliers and M / 3-1 first adders.

[0030] In some possible implementations, the two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, where N is an integer greater than 2.

[0031] In the second aspect, the present application provides a data processing method, including: obtaining an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the first initial parameter is K, K is a non-zero number, and the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; obtaining the initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with the initial convolution kernel of the Lth layer; expanding each parameter in the initial convolution kernel of the L+1th layer by K times; determining the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion processing. Wherein, K can be a positive integer.

[0032] In the method, the parameters in the initial convolution kernel of the Lth layer of the neural network are processed so that when the convolution processing is performed according to the equivalent convolution kernel obtained by the processing, one multiplication operation can be reduced. Therefore, when the number of parameters in the initial convolution kernel is one more than the number of multipliers in the device for performing the convolution calculation, there is no need to use at least one more device to cause other multipliers and adders in the device to run idle, thereby saving resources and improving resource utilization.

[0033] In addition, although the equivalent convolution kernel is reduced by K times during the processing of the initial convolution kernel, the problem that the convolution result obtained by using the equivalent convolution kernel for convolution processing is K times smaller than the convolution result obtained by using the initial convolution kernel for convolution processing can be solved by expanding the corresponding parameters in the L+1 layer of the neural network.

[0034] Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, and so on.

[0035] In some possible implementations, the equivalent convolution kernel of the Lth layer includes M second parameters and a first parameter, the M second parameters respectively correspond to M second eigenvalues ​​of the feature map, the first parameter corresponds to the first eigenvalue of the feature map, the first parameter is 1, and the convolution processing on the feature map of the Lth layer includes: performing multiplication and accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain a multiplication and accumulation result; and performing an addition operation on the multiplication and accumulation result and the first eigenvalue.

[0036] That is to say, the parameters in the equivalent convolution kernel can be obtained by dividing all the parameters in the initial convolution kernel by one of the non-zero first initial parameters. At this time, the first parameter corresponding to the first initial parameter in the equivalent convolution kernel is 1. In this way, when the feature map is processed using the equivalent convolution kernel, the multiplication and accumulation unit can be used only when the M second parameters other than the first parameter in the equivalent convolution kernel are calculated to multiply and accumulate the corresponding M eigenvalues. Then, after the multiplication and accumulation results of the M second parameters and the M eigenvalues ​​are calculated, the multiplication and accumulation results can be added to the first eigenvalue corresponding to the first parameter, thereby obtaining the convolution result of the equivalent convolution kernel and the feature map.

[0037] In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, including: when the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer reduced by m times and the first initial parameter is not greater than the first threshold. Wherein, m can be a positive integer.

[0038] The data processing device in this implementation can first reduce all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is greater than the first threshold, so that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than the first threshold. The first threshold can be less than or equal to the maximum expressible value of the processor, or the first threshold can be less than or equal to the maximum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the maximum expressible value range of the data processing device.

[0039] Optionally, the adding operation on the multiplication-accumulation result and the first eigenvalue includes: reducing the first eigenvalue by m times; and adding the multiplication-accumulation result and the reduced first eigenvalue.

[0040] In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, including: when the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is enlarged by n times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer enlarged by n times and the first initial parameter is not less than the second threshold. Wherein, n can be a positive integer.

[0041] The data processing device in this implementation can first expand all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is less than the second threshold value, so that the quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold value, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than the second first threshold value. The second threshold value can be greater than or equal to the minimum expressible value of the processor, or the second threshold value can be greater than or equal to the minimum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the minimum expressible value range of the data processing device.

[0042] Optionally, the adding operation on the multiplication-accumulation result and the first eigenvalue includes: enlarging the first eigenvalue by n times; and adding the multiplication-accumulation result and the enlarged first eigenvalue.

[0043] In a third aspect, the present application provides a data processing device, comprising: a processing module, used to obtain an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the first initial parameter is K, K is a non-zero number, and the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; an acquisition module, used to obtain the initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with the initial convolution kernel of the Lth layer; an expansion module, used to expand each parameter in the initial convolution kernel of the L+1th layer by K times; a determination module, used to determine the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion process. Wherein, K can be a positive integer.

[0044] Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, and so on.

[0045] In some possible implementations, the equivalent convolution kernel of the Lth layer includes M second parameters and one first parameter, the M second parameters respectively correspond to M second eigenvalues ​​of the feature map, the first parameter corresponds to the first eigenvalue of the feature map, and the first parameter is 1; wherein, the convolution processing on the feature map of the Lth layer includes: performing multiplication and accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain a multiplication and accumulation result; and performing an addition operation on the multiplication and accumulation result and the first eigenvalue.

[0046] In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, including: when the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer reduced by m times and the first initial parameter is not greater than the first threshold. Wherein, m can be a positive integer.

[0047] Optionally, the adding operation on the multiplication-accumulation result and the first eigenvalue includes: reducing the first eigenvalue by m times; and adding the multiplication-accumulation result and the reduced first eigenvalue.

[0048] In some possible implementations, the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, including: when the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is enlarged by n times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer enlarged by n times and the first initial parameter is not less than the second threshold. Wherein, n can be a positive integer.

[0049] Optionally, the adding operation on the multiplication-accumulation result and the first eigenvalue includes: enlarging the first eigenvalue by n times; and adding the multiplication-accumulation result and the enlarged first eigenvalue.

[0050] In a fourth aspect, the present application provides a data device, comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to execute the method in the second aspect.

[0051] In a fifth aspect, the present application provides a computer-readable medium storing instructions for execution by a device, wherein the instructions are used to implement the method in the second aspect.

[0052] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the second aspect.

[0053] In a seventh aspect, the present application provides a chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method in the second aspect.

[0054] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the second aspect.

[0055] In an eighth aspect, the present application provides a computing device, comprising a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method in the second aspect.

[0056] In a ninth aspect, the present application provides a data processing device, comprising a programmable device and a memory, wherein the memory is used to store configuration files required for the operation of the programmable device, and the programmable device is used to read the configuration files from the memory and execute them to implement the method of the second aspect.

[0057] Optionally, as an implementation manner, the programmable device includes a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the structure of a convolutional neural network provided for this application.

[0059] Figure 2 A schematic diagram of the structure of another convolutional neural network provided for this application.

[0060] Figure 3 A schematic diagram of the hardware structure of a chip provided in this application.

[0061] Figure 4 A schematic diagram of the structure of a system architecture provided for this application.

[0062] Figure 5 A flowchart of a data processing method provided in this application.

[0063] Figure 6 A schematic diagram of a method for obtaining an equivalent convolution kernel provided in this application.

[0064] Figure 7 A schematic diagram of a method for obtaining equivalent parameters provided in this application.

[0065] Figure 8 A schematic diagram of another method for obtaining equivalent parameters provided in this application.

[0066] Fig. 9 A schematic diagram of another method for obtaining equivalent parameters provided in this application.

[0067] Fig.10 A schematic diagram of another method for obtaining an equivalent convolution kernel provided in this application.

[0068] Fig.11 A schematic diagram of another method for obtaining an equivalent convolution kernel provided in this application.

[0069] Fig.12 A schematic diagram of the structure of a data processing device provided in this application.

[0070] Fig.13 A schematic diagram of the structure of another data processing device provided in this application.

[0071] Fig.14 A schematic diagram of the structure of another data processing device provided in this application.

[0072] Fig.15A schematic diagram of the structure of another data processing device provided in this application.

[0073] Fig.16 A schematic diagram of a method for reading data provided in this application.

[0074] Fig.17 A schematic diagram of a method for reading data provided in this application.

[0075] Fig.18 A schematic diagram of the structure of another data processing device provided in this application.

[0076] Fig.19 A schematic diagram of the structure of another data processing device provided in this application.

[0077] Fig. 20 A structural schematic diagram of a data processing device provided in this application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application are described below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0079] The embodiments of the present application involve related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and other related concepts of the neural network that may be involved in the embodiments of the present application are first introduced below.

[0080] Convolutional neural network is a deep neural network with convolution structure. Convolutional neural network contains a feature extractor composed of convolution layer and subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolution layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangular arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0081] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0082] Combine the following Figure 1 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0083] The structure of the convolutional neural network in the embodiment of the present application can be as follows: Figure 1 As shown. Figure 1 In the figure, the convolutional neural network (CNN) 300 may include an input layer 310, a convolutional layer / pooling layer 320 (wherein the pooling layer is optional), and a neural network layer 330.

[0084] Taking image processing as an example (the operation is similar when the input data is text or voice), the input layer 310 can obtain the image to be processed, and hand over the acquired image to be processed to the convolution layer / pooling layer 320 and the subsequent neural network layer 330 for processing, and the image processing result can be obtained.

[0085] Below Figure 1 The internal layer structure of CNN 300 is introduced in detail.

[0086] Convolutional layer / pooling layer 320:

[0087] Convolutional Layer:

[0088] like Figure 1 The convolution layer / pooling layer 320 shown may include layers 321-326 as shown in the examples. For example, in one implementation, layer 321 is a convolution layer, layer 322 is a pooling layer, layer 323 is a convolution layer, layer 324 is a pooling layer, layer 325 is a convolution layer, and layer 326 is a pooling layer; in another implementation, layers 321 and 322 are convolution layers, layer 323 is a pooling layer, layers 324 and 325 are convolution layers, and layer 326 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0089] The following will take the convolution layer 321 as an example, and take the input data as an image as an example to introduce the internal working principle of a convolution layer. When the input data is voice or text or other types of data, the internal working principle of the convolution layer is similar.

[0090] The convolution layer 321 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator may be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0091] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions.

[0092] When the convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (for example, 321) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 300 increases, the features extracted by the later convolutional layers (for example, 326) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0093] Pooling layer:

[0094] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 1The layers 321-326 illustrated in 320 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. For example, in the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0095] Neural network layer 330:

[0096] After being processed by the convolution layer / pooling layer 320, the convolution neural network 300 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 300 needs to use the neural network layer 330 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 330 may include multiple hidden layers (such as Figure 3 As shown in 331, 332 to 33n) and the output layer 340, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0097] After the multiple hidden layers in the neural network layer 330, that is, the last layer of the entire convolutional neural network 300 is the output layer 340, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 300 (such as Figure 1 The propagation from 310 to 340 is forward propagation), and the reverse propagation (such as Figure 1 The propagation from 340 to 310 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 300 and the error between the result output by the convolutional neural network 300 through the output layer and the ideal result.

[0098] The structure of the neural network in the embodiment of the present application can be as follows Figure 2 As shown. Figure 2 In FIG. 4 , a convolutional neural network (CNN) 400 may include an input layer 410, a convolutional layer / pooling layer 420 (where the pooling layer is optional), and a neural network layer 430. Figure 1 compared to, Figure 2 The convolutional layer / pooling layer 420 in the embodiment of the present invention is implemented in parallel, and the features extracted are input to the full neural network layer 430 for processing. The neural network layer 430 may include multiple hidden layers, i.e., hidden layer 1 to hidden layer n, which may be recorded as 431 to 43n.

[0099] It should be noted that Figure 1 and Figure 2 The convolutional neural network shown is only an example of two possible convolutional neural networks in an embodiment of the present application. In specific applications, the convolutional neural network in the embodiment of the present application may also exist in the form of other network models.

[0100] like Figure 1 and Figure 2 The algorithms or operators of each layer in the convolutional neural network shown in the figure can be used as follows: Figure 3 is implemented in the chip shown.

[0101] Figure 3 The hardware structure of a chip provided in an embodiment of the present application includes a neural network processor 50. The chip can be arranged in Figure 4 The chip can also be set in the client device 240 shown in FIG. Figure 4 The training device 220 shown is used to complete the training work of the training device 220 and output the target model 201.

[0102] The neural network processor NPU 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0103] In some implementations, the computing circuit 503 includes multiple processing engines (PEs) inside.

[0104] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of matrix C is stored in the accumulator 508.

[0105] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0106] In some implementations, the vector calculation unit can 507 store the vector of processed outputs to the unified memory 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

[0107] The unified memory 506 is used to store input data and output data.

[0108] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0109] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0110] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0111] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0112] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0113] in, Figure 1 and Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0114] like Figure 4 As shown, the embodiment of the present application provides a system architecture 200. Figure 4 In the example, the data acquisition device 260 is used to acquire training data. Taking the target model 201 for image processing as an example, the training data may include training images and classification results corresponding to the training images, wherein the results of the training images may be manually pre-labeled results. The target model 201 may also be referred to as the target rule 201.

[0115] After collecting the training data, the data collection device 260 stores the training data in the database 230 , and the training device 220 obtains the target model 201 through training based on the training data maintained in the database 230 .

[0116] The following describes how the training device 220 obtains the target model 201 based on the training data. The training device 220 processes the input original image and compares the output image with the original image until the difference between the image output by the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model 201.

[0117] The target model 201 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 230 does not necessarily all come from the collection of the data acquisition device 260, and may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model 201 entirely based on the training data maintained by the database 230, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0118] The target model 201 obtained by training the training device 220 can be applied to different systems or devices, such as Figure 2 The client device 240 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., and may also be a server or a cloud, etc.

[0119] The training device 220 can generate a corresponding target model 201 based on different training data for different goals or different tasks. The corresponding target model 201 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0120] The target model 201 obtained by training with the training device 220 may be a CNN, a deep convolutional neural network (DCNN), etc.

[0121] It is worth noting that Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc., the type of training data, and the type or function of the neural network shown in the figure do not constitute any limitation. Figure 4 In the embodiment, the client device 240 and the training device 220 may be the same device.

[0122] After the training device 220 trains the neural network, if the target task is directly performed according to the trained neural network, such as image segmentation, image recognition, etc., and each PE in the computing circuit in the neural network processor can only calculate the convolution result of the convolution kernel of size M and the feature map of size M, and the number of parameters in at least one convolution kernel in the neural network is M+1, then at least two PEs are required to calculate the convolution result of the convolution kernel of size M+1 and the feature map of size M+1. However, this will cause more multipliers and adders in the PE to run idle, thereby wasting resources.

[0123] For example, if a neural network includes a depthwise separable convolution layer, the size of the convolution kernel of the depthwise separable convolution layer is 3*3*1, and one PE can only calculate 8 multiplication and accumulation operations, then two PEs are required to calculate the convolution result of the convolution kernel. This will cause 6 multipliers and 4 adders in one PE to run idle, thus wasting resources.

[0124] In order to solve the problem of resource waste, the present application proposes a new data processing method and a data processing device.

[0125] Figure 5 is a schematic flow chart of a data processing method according to an embodiment of the present application. The method at least includes S510 to S540. The method may be Figure 3 The main CPU in the program is used for execution.

[0126] S510, obtaining an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the first initial parameter is K, K is a non-zero number, and the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer.

[0127] The neural network in this embodiment can be any neural network that includes convolution processing. For example, the neural network in this embodiment can be Figure 1 or Figure 2 The convolutional neural network shown in .

[0128] The Lth layer in this embodiment may be any layer including convolution processing, for example, it may be a convolution layer, and further, it may be a depth-separable convolution layer.

[0129] The initial convolution kernel in this embodiment may be a convolution kernel obtained by training a neural network, or may be a convolution kernel obtained after training and optimizing the neural network.

[0130] In this embodiment, the parameters in the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, which can be understood as: a non-zero parameter in the initial convolution kernel of the Lth layer is called the first initial parameter, all parameters in the initial convolution kernel are divided by the first initial parameter, and the quotient of each parameter and the first initial parameter is obtained, and the equivalent convolution kernel used for convolution processing of the feature map of the Lth layer is determined according to the quotient of each parameter and the first initial parameter. That is to say, after obtaining the equivalent convolution kernel, when the Lth layer in the neural network is used to process the feature map subsequently, the initial convolution kernel is no longer used, but the equivalent convolution kernel is used.

[0131] Among them, the feature map input to the Lth layer may include the features of the image input to the neural network, or may include the voice features contained in the voice data input to the neural network, or may include the text features contained in the text data input to the neural network, and so on.

[0132] S520, obtaining an initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with an initial convolution kernel of the Lth layer.

[0133] That is to say, the initial parameters in the first layer after the Lth layer are obtained, which have a mapping relationship with the parameters in the aforementioned initial convolution kernel in the Lth layer.

[0134] The initial parameters of the L+1th layer may be the parameters of the L+1th layer obtained by training the neural network, or may be the parameters of the L+1th layer obtained after training and optimizing the neural network.

[0135] The initial parameters in the L+1th layer that have a mapping relationship with the parameters in the aforementioned initial convolution kernel in the Lth layer can be understood as follows: if the Lth layer of the neural network uses the aforementioned initial convolution kernel to process the feature map and outputs the eigenvalues ​​obtained by the convolution, and then the eigenvalues ​​obtained by the convolution are input into the L+1th layer, the parameters in the L+1th layer originally used to process these eigenvalues ​​are the initial parameters in the L+1th layer that have a mapping relationship with the parameters in the initial convolution kernel of the Lth layer.

[0136] The L+1th layer can be a depth-wise separable convolutional layer, a normal convolutional layer, or a fully connected layer, etc. Among them, the normal convolutional layer can also be called a regular convolutional layer or a standard convolutional layer. Different types of L+1th layers have different mapping relationships between their initial parameters and the initial parameters of the convolution kernel of the Lth layer.

[0137] An example of the mapping relationship between the initial parameters in the L+1th layer of the neural network and the initial parameters in the initial convolution kernel of the Lth layer will be combined in the subsequent content Figure 7 , Figure 8 and Fig. 9 Make an introduction.

[0138] S530, enlarging each parameter in the initial convolution kernel of the L+1th layer by K times.

[0139] Because in S510, the parameters in the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameters in the initial convolution kernel of the Lth layer, the parameters in the equivalent convolution kernel are generally K times smaller than the initial convolution kernel of the Lth layer. In order to ensure the accuracy of the result of the neural network processing the input data, the initial parameters of the L+1th layer can be expanded by K times, so that the characteristic values ​​output by the Lth layer can be compensated.

[0140] S540, determining an equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion process.

[0141] In this embodiment, the equivalent convolution kernel of the L+1th layer is used to process the eigenvalues ​​output by the Lth layer, or to process the eigenvalues ​​input to the L+1th layer. That is to say, when the neural network is used for business processing in an application scenario, such as image classification, image segmentation, image recognition and other business processing, the initial convolution kernel of the L+1th layer is no longer used, but the equivalent convolution kernel of the L+1th layer is used.

[0142] The data processing method in this embodiment performs relevant processing on the initial convolution kernel of the Lth layer, so that when the equivalent convolution kernel obtained by the processing is used to perform convolution processing on the feature map of the input Lth layer, the convolution operation between the equivalent convolution kernel and the feature map no longer requires all multiplication and accumulation operations, but the convolution operation between the equivalent convolution kernel and the feature map can be decomposed into multiplication and accumulation operations of part of the convolution parameters and the corresponding eigenvalues, and non-multiplication and accumulation operations of the multiplication and accumulation operation results and the eigenvalues ​​in the feature map without parameter multiplication and accumulation operations. In this way, when the Lth layer of the neural network performs convolution processing on the feature map, if the number of parameters in the convolution kernel of the Lth layer is one more than the number of multiplication and accumulation times that the operation circuit performing the convolution processing can calculate at one time, then the problem of idling most of the multipliers and adders in the multiplied operation circuit due to the need to use one more operation circuit, thereby causing a waste of resources.

[0143] In addition, when obtaining the equivalent convolution kernel based on the initial convolution kernel of the L layer, the difference between the equivalent convolution kernel and the initial convolution kernel caused by the adopted processing can be compensated by the difference between the equivalent convolution kernel of the L+1th layer and the initial convolution kernel, thereby ensuring the accuracy of the neural network's processing of the input data.

[0144] In this embodiment, as a possible implementation, obtaining an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer may include the following steps: determining a parameter in the initial convolution kernel of the Lth layer as a first initial parameter, and when the first initial parameter is not zero, dividing all parameters in the initial convolution kernel by the first initial parameter, and the convolution kernel formed by the quotient obtained is the equivalent convolution kernel. If the first initial parameter is 0, the initial convolution kernel can be directly used as the equivalent convolution kernel.

[0145] In this implementation, the value obtained by dividing the first initial parameter in the initial convolution kernel of the Lth layer by the first initial parameter is 1, that is, there is a parameter in the equivalent convolution kernel whose value is 1, and this parameter with a value of 1 is called the first parameter.

[0146] In this embodiment, assuming that the initial convolution kernel of the Lth layer includes a first initial parameter and M second initial parameters, the corresponding equivalent convolution kernel of the Lth layer includes M second parameters and one first parameter, and the M second parameters can respectively correspond to the M second eigenvalues ​​of the feature map to be processed, and the first parameter can correspond to the first eigenvalue of the feature map to be processed.

[0147] In this embodiment, in some implementations, when obtaining the parameters of the equivalent convolution kernel of the Lth layer based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the following steps may be included: when the quotient of at least one parameter in the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel is greater than a first threshold, all the parameters in the initial convolution kernel of the Lth layer may be first reduced by m times, and then the parameters reduced by m times are divided by the first initial parameter, wherein the quotient of the parameter obtained after any parameter in the initial convolution kernel of the Lth layer is reduced by m times and the first initial parameter should not be greater than the first threshold.

[0148] The first threshold value may be the maximum expressible value of the device used when performing convolution processing on the feature map based on the equivalent convolution kernel. For example, the device used when performing convolution processing on the feature map based on the equivalent convolution kernel is as follows: Figure 3 When the neural network processor 50 is shown in FIG. 1 , the first threshold value may be less than or equal to the maximum expressible value of the memory and the computing unit therein. This can avoid data overflow, thereby improving the accuracy of the neural network in business processing.

[0149] In some possible implementations, when obtaining the parameters of the equivalent convolution kernel of the Lth layer based on the quotient of the parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the following steps may be included: when the quotient of at least one parameter in the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, all the parameters of the initial convolution kernel of the Lth layer may be first expanded by n times, and then the expanded parameters are divided by the first initial parameter to obtain an equivalent convolution kernel, wherein the quotient of the parameter obtained after any parameter of the initial convolution kernel of the Lth layer is expanded by n times and the first initial parameter should not be less than the second threshold.

[0150] The second threshold value may be the minimum expressible value of the device used when performing convolution processing on the feature map based on the equivalent convolution kernel. For example, the device used when performing convolution processing on the feature map based on the equivalent convolution kernel is as follows: Figure 3 When the neural network processor 50 is shown in FIG. 1 , the second threshold value may be greater than or equal to the minimum expressible value of the memory and the computing unit therein. This can avoid numerical overflow, thereby improving the accuracy of the neural network in business processing.

[0151] The following takes the Lth layer as a depth-wise separable convolution layer and the initial convolution kernel of the Lth layer as a 3*3 convolution kernel as an example to introduce an implementation method of obtaining the equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network.

[0152] In this implementation, any parameter in the 3*3 convolution kernel in the Lth layer is used as the first initial parameter (or can be called a common factor); when the first initial parameter is not 0, all 9 parameters of the convolution kernel (i.e., 8 second initial parameters and 1 first initial parameter) are divided by the absolute value of the first initial parameter. When the first initial parameter is 0, all parameters remain unchanged to ensure that the equivalent convolution kernel has 8 normal second parameters and a second parameter with a value of 1, 0 or -1. In this way, the convolution operation of the Lth layer can be changed from 9 MAC operations to the calculation of adding 8 MAC results (i.e., 8 second parameters and 8 second eigenvalues) and a constant (i.e., the first eigenvalue). Based on this implementation, an adder can be added to the existing PE that can process 8 MACs, so that processing can be performed based on a 3*3 convolution kernel at each clock, so that the new PE can work at 100% efficiency when processing convolution processing, greatly improving the performance of the hardware.

[0153] like Figure 6 As shown in the figure, when the initial convolution kernel includes nine initial parameters w0, w1, w2, w3, w4, w5, w6, w7 and w8, w4 is selected as the first initial parameter; when w4 is greater than 0, these nine initial parameters are divided by w4 to obtain an equivalent convolution kernel. In this case, w4 is called a common factor; when w4 is equal to 0, no additional processing is performed on these nine initial parameters, that is, the equivalent convolution kernel is the same as the initial convolution kernel. In this case, the common factor is 0; when w4 is less than 0, these nine initial parameters are divided by -w4 to obtain an equivalent convolution kernel. In this case, -w4 is called a common factor.

[0154] Optionally, when w4 is less than 0, the nine initial parameters may be divided by w4 instead of -w4.

[0155] The equivalent convolution kernel obtained according to the initial convolution kernel is used to perform convolution processing on the eigenvalues ​​of the Lth layer input, and the convolution result is used as the input of the L+1th layer.

[0156] The initial convolution kernel of the Lth layer is introduced below: Figure 6 When the convolution kernel shown in , and the L+1th layer is also a depth-separable convolution layer, an implementation method for obtaining the equivalent parameters of the L+1th layer according to the initial parameters of the L+1th layer of the neural network.

[0157] When the L+1th layer is a depth-separable convolution layer, if the common factor of the initial convolution kernel of the Lth layer is 0, no additional processing is required for the initial convolution kernel of the L+1th layer, that is, the initial convolution kernel of the L+1th layer is an equivalent convolution kernel; otherwise, the initial convolution kernel of the L+1th layer can be multiplied by the common factor of the initial convolution kernel of the Lth layer and processed. Figure 6 The initial convolution kernel of the L+1th layer is the convolution kernel in the L+1th layer corresponding to the initial convolution kernel of the Lth layer. An example of the initial convolution kernel of the L+1th layer and the equivalent convolution kernel is as follows: Figure 7 shown.

[0158] Depend on Figure 7 It can be seen that the second parameter in the equivalent convolution kernel of the L+1th layer is equal to the quotient of the product of the corresponding second initial parameter in the initial convolution kernel of the L+1th layer multiplied by the first initial parameter in the Lth layer and the first initial parameter in the initial convolution kernel of the L+1th layer.

[0159] The following describes a method for obtaining equivalent parameters of the L+1 layer according to the initial parameters of the L+1 layer of the neural network when the initial convolution kernel of the Lth layer is a depth-wise separable convolution kernel and the L+1th layer is a normal convolution layer.

[0160] like Figure 8 As shown in the figure, the Lth layer has 16 input channels, which are numbered from 0 to 15. These 16 input channels correspond one to one to 16 initial convolution kernels, and the common factors of these 16 initial convolution kernels are w0 to w15. When obtaining the equivalent parameters of the L+1th layer according to the initial parameters of the L+1th layer of the neural network, each convolution kernel in the input channel of the L+1th layer can be multiplied by the corresponding common factors from w0 to w15 to obtain the equivalent parameters of the L+1th layer.

[0161] The following describes a method for obtaining equivalent parameters of the L+1 layer according to the initial parameters of the L+1 layer of the neural network when the initial convolution kernel of the Lth layer is a depth-wise separable convolution kernel and the L+1th layer is a fully connected layer.

[0162] like Fig. 9 As shown in the figure, when L+1 is a fully connected layer, it is first necessary to clarify the position of each eigenvalue in the feature map output by the L+1th layer after the feature map is converted into a one-dimensional vector. Based on the characteristics that the one-dimensional vector of the feature map is of the same length as the parameters of the fully connected layer and their positions are in a corresponding relationship, it is identified that each parameter of the L+1th layer corresponds to which initial convolution kernel in the Lth layer, and each parameter is multiplied by the common factor of the corresponding initial convolution kernel, where the common factor is not zero.

[0163] In the data processing method of the present application, all the depth-separable convolutional layers in the neural network can be performed Figure 5All or part of the steps in can also be performed on only some of the depthwise separable convolutional layers in the neural network. Figure 5 All or part of the steps in .

[0164] The following takes the first threshold as the maximum expressible value of the built-in hardware data format of the device for implementing convolution processing, that is, the quotient of any second initial parameter in the initial convolution kernel of the Lth layer and the first initial parameter exceeds the maximum expressible value range, and the maximum expressible value is 2 to the 16th power as an example to introduce how to process the initial convolution kernel of the Lth layer to obtain an equivalent convolution kernel that does not exceed the expressible value range.

[0165] like Fig.10 As shown in the middle left figure, the first initial parameter of the initial convolution kernel of the Lth layer is 0.001, and one of the second initial parameters is 64000. If 64000 is directly divided by 0.001, the quotient will be greater than the maximum value. Fig.10 As shown in the middle figure in the figure, all initial parameters can be divided by 2 to the power of 10 (or multiplied by 2 to the negative power of 10) and then divided by 0.001 to obtain the following Fig.10 The equivalent convolution kernel shown in the middle right figure. Among them, 2 to the power of -10 is the number closest to the power of 2 that is smaller than the smallest expressible value.

[0166] The following takes the second threshold as the minimum expressible value of the built-in hardware data format of the device for implementing convolution processing, that is, the quotient of any second initial parameter and the first initial parameter in the initial convolution kernel of the Lth layer exceeds the minimum expressible value range, and the minimum expressible value is 2 to the power of -16 as an example to introduce how to process the initial convolution kernel of the Lth layer to obtain an equivalent convolution kernel that does not exceed the expressible value range.

[0167] like Fig.11 As shown in the middle left figure, when the first initial parameter of the initial convolution kernel of layer L is 64000 and one of the second initial parameters is 0.001, if 0.001 is directly divided by 64000, the quotient will be less than the maximum value. Fig.11 As shown in the middle figure, all initial parameters can be multiplied by 2 to the 16th power and divided by 64000 to obtain Fig.11 The equivalent convolution kernel is shown in the middle right figure. Among them, 2 to the power of 16 is the number closest to the power of 2 that is larger than the maximum expressible value.

[0168] The data processing device provided by the present application is introduced below. The data processing device provided by the present application can be used to perform convolution processing on a to-be-convolved matrix according to a two-dimensional convolution kernel, wherein the two-dimensional convolution kernel includes a first parameter and M second parameters, and the to-be-convolved matrix includes a first eigenvalue and M second eigenvalues, wherein the M second parameters correspond one-to-one to the M second eigenvalues, and the first parameter corresponds to the first eigenvalue.

[0169] Fig.12 This is a schematic structural diagram of the data processing device of the present application. The data processing device includes: M multipliers, M-1 first adders and a second adder, the M multipliers are Mul 0 to Mul M, the M-1 first adders are ADD 0 to ADD M-2, and the second adder is ADD M-1. M is an even number.

[0170] The M multipliers and the M-1 first adders are used to perform multiplication-accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain a multiplication-accumulation result. The second adder is used to perform an addition operation on the multiplication-accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved.

[0171] When the data processing device performs convolution processing on a matrix to be convolved of size M+1 according to a convolution kernel of size M+1, the convolution result of M parameters in the convolution kernel and M eigenvalues ​​in the matrix to be convolved is calculated by M multipliers and M-1 adders, and the sum of the convolution result and the M+1th eigenvalue in the matrix to be convolved is calculated by another adder, and finally the sum is used as the convolution result of the convolution kernel on the matrix to be convolved. This makes it possible to avoid using a plurality of traditional data processing devices including M+1 multipliers and M adders for calculation when the size of the convolution kernel that needs to be convolved is one more than the number of multiplication and accumulation units in the data processing device, thereby avoiding waste of resources and improving resource utilization.

[0172] For example, Fig.12 The data processing device shown can be used to: perform convolution processing on the input L-th layer eigenvalue matrix to be convolved based on the equivalent convolution kernel of the L-th layer in S510.

[0173] Fig.12 The data processing device shown may be Figure 3 A component of the computing circuit 503 in the neural network processor 50.

[0174] In some possible implementations, the first parameter is equal to 1. That is, the first parameter corresponding to the first eigenvalue input to the second adder is 1. In this way, the accuracy of the convolution result calculated by the data processing device can be guaranteed.

[0175] In some possible implementations, the device further includes a processor, the processor being used to: determine the two-dimensional convolution kernel according to an initial convolution kernel, the initial convolution kernel including a first initial parameter and M second initial parameters, the M second parameters corresponding to the M second initial parameters one-to-one, the first parameter corresponding to the first initial parameter, the second parameter being equal to the quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter being not zero. In other words, the processor can be used to execute S510.

[0176] When the data processing device in this implementation is used to calculate the convolution result of the initial convolution kernel and the matrix to be convolved, all parameters in the initial convolution kernel can be divided by one of the non-zero parameters (i.e., the first initial parameter) through the processor, so that one parameter (i.e., the first parameter) in the obtained two-dimensional convolution kernel can be 1. In this way, when the above-mentioned M multipliers and M adders are used to calculate the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, a more accurate value can be obtained.

[0177] Optionally, the processor can also be used to: determine the convolution result of the initial convolution kernel and the matrix to be convolved based on the convolution result of the two-dimensional convolution kernel and the matrix to be convolved, wherein the convolution result of the initial convolution kernel and the matrix to be convolved is equal to the product of the convolution result of the two-dimensional convolution kernel and the matrix to be convolved and the first initial parameter.

[0178] In some possible implementations, when the quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, before determining the two-dimensional convolution kernel based on the initial convolution kernel, the processor is further used to: reduce the M second parameters or the M second initial parameters by m times, wherein the quotient of any second initial parameter reduced by m times and the first initial parameter is not greater than the first threshold.

[0179] The data processing device in this implementation can first reduce all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is greater than the first threshold, so that the quotient of any second initial parameter after reduction and the first initial parameter is not greater than the first threshold, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not greater than the first threshold. The first threshold can be less than or equal to the maximum expressible value of the processor, or the first threshold can be less than or equal to the maximum expressible value of the multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the maximum expressible value range of the data processing device.

[0180] The operation of reducing the first eigenvalue by m times may be performed by the processor or by a shifter. For example, when m is 2 raised to the power of s, the first eigenvalue may be reduced by m times by a shifter capable of at least shifting left by s bits, where s is a non-negative integer. An exemplary structural diagram of an apparatus for performing convolution processing on a feature map based on an equivalent convolution kernel in this scenario is shown in FIG. Fig.13 shown.

[0181] pass Fig.13 When the device performs convolution processing, the M second parameters in the equivalent convolution kernel and the M second eigenvalues ​​in the feature map can be input into the M multipliers, and the first eigenvalue in the feature map can be input into the shifter for shifting. The multiplication and accumulation results obtained by the multiplication and accumulation processing of the M second parameters and the M second eigenvalues ​​by the M multipliers and the M adders therein, and the eigenvalue obtained by shifting the first eigenvalue left by m times by the shifter are input into ADD M-1, and the sum output by ADD M-1 is the convolution result of the equivalent convolution kernel and the feature map.

[0182] This is because when the second initial parameter or the second parameter is reduced by m times, the first parameter should also be reduced by m times accordingly, so the processing result of the first parameter on the first eigenvalue should also be reduced by m times. The data processing device of the present application directly reduces the first eigenvalue by m times to ensure the accuracy of the convolution result.

[0183] For example, based on Fig.10 When the equivalent convolution kernel shown in the right figure performs convolution processing on the feature map, the first eigenvalue in the feature map can be input into the shifter and shifted left by 10 bits, and the obtained eigenvalue is added to the multiplication and accumulation result to obtain the convolution result.

[0184] In some possible implementations, when the quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, before determining the two-dimensional convolution kernel based on the initial convolution kernel, the processor is further used to: expand the M second parameters or the M second initial parameters by n times, wherein the quotient of any second initial parameter expanded by n times and the first initial parameter is not less than the second threshold.

[0185] The data processing device in this implementation can first expand all the second initial parameters by m times when the quotient of any second initial parameter in the initial convolution kernel and the first initial parameter is less than the second threshold value, so that the quotient of any second initial parameter after expansion and the first initial parameter is not less than the second threshold value, so that the second parameter obtained by dividing the reduced second initial parameter by the first initial parameter is not less than the second first threshold value. The second threshold value can be greater than or equal to the minimum expressible value of the processor, or the second threshold value can be greater than or equal to the minimum expressible value of the above-mentioned multiplier and adder in the data processing device. In this way, the calculated convolution result will not cause the problem of inaccurate calculation results due to overflowing the minimum expressible value range of the data processing device.

[0186] The operation of expanding the first eigenvalue by n times can be performed by the main CPU to which the neural network processor performing the convolution processing is attached, or can be performed by a shifter. For example, when n is 2 to the tth power, the first eigenvalue can be expanded by n times by a shifter capable of at least right shifting t bits, where n is a non-negative integer. An exemplary structural diagram of a device for performing convolution processing on a feature map based on an equivalent convolution kernel in this scenario is shown in FIG. Fig.13 shown.

[0187] pass Fig.13 When the device performs convolution processing, the M second parameters in the equivalent convolution kernel and the M second eigenvalues ​​in the feature map can be input into the M multipliers, and the first eigenvalue in the feature map can be input into the shifter for shifting. The multiplication and accumulation results obtained by the multiplication and accumulation processing of the M second parameters and the M second eigenvalues ​​by the M multipliers and the M adders therein, and the eigenvalue obtained by right-shifting the first eigenvalue by n times by the shifter are input into ADD M-1, and the sum output by ADD M-1 is the convolution result of the equivalent convolution kernel and the feature map.

[0188] This is because when the second initial parameter or the second parameter is enlarged by m times, the first parameter should also be enlarged by m times accordingly, so the processing result of the first parameter on the first eigenvalue should also be enlarged by m times. The data processing device of the present application directly enlarges the first eigenvalue by m times to ensure the accuracy of the convolution result.

[0189] For example, based on Fig.11 When the equivalent convolution kernel shown in the right figure performs convolution processing on the feature map, the first eigenvalue in the feature map can be input into the shifter and shifted right by 16 bits, and the obtained eigenvalue is added to the multiplication and accumulation result to obtain the convolution result.

[0190] In some possible implementations, the processor includes a combination of one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU).

[0191] In some possible implementations, M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix. Optionally, the M multipliers and the M-1 first adders may constitute a product-accumulator.

[0192] For example, Fig.14 As shown, the data processing device may include 8 multipliers and 8 adders, wherein 8 multipliers and 7 adders are used for multiplying and accumulating 8 convolution kernel parameters and 8 eigenvalues, and another adder is used for adding the multiplication and accumulation results and another eigenvalue, thereby obtaining the convolution result of the two-bit convolution kernel and the eigenvalue matrix to be convolved.

[0193] It is understandable that the data processing device can be based on the PE that processes 8 multiplication and accumulation operations, and an adder can be added to obtain the data processing device of the present application. In other words, the data processing device of the present application can reuse the existing hardware logic, and then add an adder to obtain the data processing device of the present application.

[0194] or, Fig.14 The 8 multipliers and 7 adders in the MCU can be composed of two PEs that can handle 4 multiplication and accumulation.

[0195] Or, if Fig.15 As shown, we can add an adder to the PE that processes 16 multiplication and accumulation operations to obtain two Fig.14 The data processing device shown in FIG. 1 is a data processing device in which the output of ADD 6 is no longer output to ADD 14 but to the newly added ADD 15.

[0196] In some possible implementations, M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix. Optionally, the M multipliers and the M-1 first adders constitute three multiplication accumulators, wherein each of the multiplication accumulators includes M / 3 multipliers and M / 3-1 first adders.

[0197] For example, three adders can be added based on three PEs that can process eight multiplications and accumulations to obtain a data processing device that can process a 5*5 feature matrix to be convolved based on a 5*5 two-bit convolution kernel in the present application. The three newly added adders are used to add the multiplication and accumulation results of the three PEs to the first eigenvalue in the feature matrix to be convolved.

[0198] In some implementations, Fig.14The device shown may further include a shifter, the output port of the shifter is connected to an input port of ADD 7; or, Fig.15 The device shown may also include two shifters, wherein the output port of one shifter is connected to the input port of ADD 15, and the output port of the other shifter is connected to the input port of ADD 14. In this case, the first eigenvalue in the feature matrix to be convolved is first input to the shifter for shifting, and the shifted result and the multiplication and accumulation result are input to the adder for addition, thereby obtaining the convolution result. This device including a shifter is used when performing convolution processing based on an equivalent convolution kernel obtained by enlarging or reducing the initial convolution kernel. Of course, when performing convolution processing based on an equivalent convolution kernel obtained without enlarging or reducing the initial convolution kernel, a device including a shifter can also be used, but in this case, the shifter does not need to shift the input first eigenvalue.

[0199] In some possible implementations of this embodiment, the two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, where N is an integer greater than 2.

[0200] The following example takes a PE that can perform convolution on a feature map containing 9 eigenvalues ​​based on a convolution kernel containing 9 parameters. Fig.16 and Fig.17 The data reading method proposed in this application is introduced as an example. For example, the PE is Fig.14 or Fig.15 Further, for example, the PE is obtained by adding an adder and even a shifter based on the PE capable of processing 8 multiplication and accumulation operations.

[0201] like Fig.16 As shown in the figure, the solid box on the left represents a feature map, a square dotted border in the solid box on the left represents a convolution window, the convolution window moves to the right, and the moving step is 1. The size of the convolution window is determined by the size of the convolution kernel represented by the solid box on the right. The first row of eigenvalues ​​in the feature map is represented by A0 to An, the second row of eigenvalues ​​is represented by B0 to Bn, and so on; the solid box in the middle represents a PE; the solid box on the right represents the convolution kernel K, an example of which is the aforementioned equivalent convolution kernel, and the size of the convolution kernel is 3*3. Fig.16 16 PEs are given as an example, namely PE0 to PE15. For example, a neural network processor includes such 16 PEs. Fig.16 In the figure, arrows indicate the direction of data flow.

[0202] Among them, each PE can convolve the feature value in a convolution kernel window based on a convolution kernel, so within one clock cycle, 16 PEs can convolve the feature values ​​in 16 convolution kernel windows based on the same convolution kernel.

[0203] like Fig.17 As shown, at the initial clock, the direct-connected cache directly connected to the unit performing convolution processing can read 3*(16+2) data from the lower-level cache. These 3*(16+2) data are A0 to A17, B0 to B17 and C0 to C17 respectively. Then the direct-connected cache can transfer A0 to A2, B0 to B2, C0 to C2 to PE0, and transfer A1 to A3, B1 to B3 and C1 to C3 to PE1, and so on, until A15 to A17, B15 to B17 and C15 to C17 are output to PE15.

[0204] At the next clock, the direct-connected cache can read only (16+2) data from the lower-level cache. These 18 data are D0 to D17, and then B0 to B2, C0 to C2, and D0 to D2 are transmitted to PE0, and B1 to B3, C1 to C3, and D1 to D3 are transmitted to PE1, and so on, until B15 to B17, C15 to C17, and D15 to D17 are transmitted to PE15.

[0205] This is because the data that the direct-connected cache needs to transmit to PE0 to PE15 in two clock cycles partially overlaps, so the direct-connected cache can reuse the repeated data read from the lower-level cache in the previous clock cycle in the next clock cycle, and only needs to read the part of the data that needs to be updated. This can save transmission resources and improve transmission efficiency.

[0206] Understandably, Fig.17 The transmission of parameters in the convolution kernel is not shown.

[0207] If a PE consists of two Fig.14 or Fig.15 For example, the PE is based on a PE capable of processing 16 multiplication and accumulation operations and an adder is added, and even a shifter is added. The method of reading data from the direct-connected cache is the same as that of Fig.17 The difference is that in the initial clock cycle, the direct cache needs to read 16 more convolutional windows in the horizontal dimension of the feature map, or in the vertical dimension of the feature map from Fig.16 In the next row of the convolution window, 16 more data are read. That is, in the initial clock cycle, the direct-connected cache needs to read 4*(16+2) data, and only 2*(16+2) data can be updated in each subsequent clock cycle.

[0208] Fig.18 1800 is an exemplary structural diagram of the data processing device of the present application. The device 1800 includes a processing module 1810, an acquisition module 1820, an expansion module 1830 and a determination module 1840. The device 1800 can implement the aforementioned Figure 5 The method shown.

[0209] For example, the processing module 1810 may be used to execute S510 , the acquiring module 1820 may be used to execute S520 , the expanding module 1830 may be used to execute S530 , and the determining module 1840 may be used to execute S540 .

[0210] In some possible implementations, the device 1800 may be Figure 3 In some other possible implementations, the device 1800 may be Figure 4 The training device 220 shown in FIG. 1 ; in some other possible implementations, the apparatus 1800 may be Figure 4 The client device 240 described in .

[0211] The present application also provides a Fig.19 The device 1900 shown includes a processor 1902, a communication interface 1903, and a memory 1904. An example of the device 1900 is a chip. Another example of the device 1900 is a computing device.

[0212] The processor 1902, the memory 1904 and the communication interface 1903 can communicate with each other via a bus. The memory 1904 stores executable code, and the processor 1902 reads the executable code in the memory 1904 to execute the corresponding method. The memory 1904 may also include an operating system and other software modules required for running processes. The operating system may be LINUX TM , UNIX TM , WINDOWS TM wait.

[0213] For example, the executable code in memory 1904 is used to implement Figure 5 The method shown in the figure, the processor 1902 reads the executable code in the memory 1804 to execute Figure 5 The method shown.

[0214] The processor 1902 may be a central processing unit (CPU) or a graphics processing unit (GPU). The memory 1904 may include a volatile memory (volatile memory), such as a random access memory (random access memory, RAM). The memory 1904 may also include a non-volatile memory (non-volatile memory, NVM), such as a read-only memory (read-only memory, ROM), a flash memory, a hard disk drive (hard disk drive, HDD) or a solid state disk (solid state disk, SSD).

[0215] The present application also provides a Fig. 20 The data processing device 2000 shown includes a programmable device 2001 and a memory 2002, wherein the memory 2002 is used to store configuration files required for the operation of the programmable device 2001, and the programmable device 2001 is used to read the configuration files from the memory 2002 and execute them to implement the corresponding method.

[0216] The programmable device may include a Field Programmable Gate Array (FPGA) or a Complex Programmable Logic Device (CPLD).

[0217] Taking FPGA as an example, those skilled in the art can understand that its basic working principle is to change the content of the configuration RAM inside the FPGA by loading a configuration data (for example, in the form of a configuration file), thereby changing the configuration of various logic resources inside the FPGA to achieve different circuit functions, and the configuration data can be loaded multiple times, so that the FPGA can complete different functions by loading different configuration data, which has good flexibility. In this practical application, it is often necessary to update the function of the FPGA. At this time, the new configuration data can be loaded into the FPGA configuration memory in advance, and then the FPGA is allowed to load the new configuration data to achieve the function defined by the new configuration data. This process is called the FPGA upgrade process. At the same time, when the FPGA leaves the factory, it will come with a configuration loading circuit for loading configuration data. The configuration loading circuit can be used to ensure the most basic loading operation after the user-defined circuit function (i.e., the function defined by the configuration data) fails.

[0218] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0219] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0220] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0223] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0224] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing device, characterized in that: The data processing device is used to perform convolution processing on a to-be-convolved matrix according to a two-dimensional convolution kernel, wherein the two-dimensional convolution kernel includes a first parameter and M second parameters, the to-be-convolved matrix includes a first eigenvalue and M second eigenvalues, the M second parameters correspond to the M second eigenvalues ​​one-to-one, the first parameter corresponds to the first eigenvalue, and the data processing device includes: M multipliers and M-1 first adders, used to perform multiplication-accumulation operations on the M second parameters and the M second eigenvalues ​​to obtain multiplication-accumulation results; A second adder is used to add the multiplication and accumulation result and the first eigenvalue to obtain a convolution result of the two-dimensional convolution kernel and the matrix to be convolved.

2. The device according to claim 1, characterized in that The first parameter is equal to 1.

3. The device according to claim 1 or 2, characterized in that The device also includes a processor, which is used to: determine the two-dimensional convolution kernel according to an initial convolution kernel, the initial convolution kernel includes a first initial parameter and M second initial parameters, the M second parameters correspond one-to-one to the M second initial parameters, the first parameter corresponds to the first initial parameter, the second parameter is equal to the quotient of the second initial parameter corresponding to the second parameter and the first initial parameter, and the first initial parameter is not zero.

4. The device according to claim 3, characterized in that When a quotient of at least one of the second initial parameters and the first initial parameter is greater than a first threshold, before determining the two-dimensional convolution kernel according to the initial convolution kernel, the processor is further configured to: The M second parameters or the M second initial parameters are reduced by m times, wherein a quotient of any second initial parameter reduced by m times and the first initial parameter is not greater than the first threshold.

5. The device according to claim 4, characterized in that Before performing the addition operation on the multiplication-accumulation result and the first eigenvalue, the processor is further configured to: Reduce the first eigenvalue by m times; Correspondingly, the second adder is specifically used to perform an addition operation on the multiplication-accumulation result and the reduced first eigenvalue.

6. The device according to claim 3, characterized in that When a quotient of at least one of the second initial parameters and the first initial parameter is less than a second threshold, before determining the two-dimensional convolution kernel according to the initial convolution kernel, the processor is further configured to: The M second parameters or the M second initial parameters are enlarged n times, wherein a quotient of any second initial parameter enlarged n times and the first initial parameter is not less than the second threshold.

7. The device according to claim 6, characterized in that Before performing the addition operation on the multiplication-accumulation result and the first eigenvalue, the processor is further configured to: Enlarging the first eigenvalue by n times; Correspondingly, the second adder is specifically used to perform an addition operation on the multiplication-accumulation result and the enlarged first eigenvalue.

8. The device according to claim 3, characterized in that The processor includes one or more of the following combinations: a central processing unit (CPU), a graphics processing unit (GPU) or a neural network processing unit (NPU).

9. The device according to claim 1 or 2, characterized in that: M is equal to 8, and the two-dimensional convolution kernel is a 3*3 matrix.

10. The device according to claim 8, characterized in that The M multipliers and the M-1 first adders constitute a product accumulator.

11. The device according to claim 1 or 2, characterized in that M is equal to 24, and the two-dimensional convolution kernel is a 5*5 matrix.

12. The device according to claim 11, characterized in that The M multipliers and the M-1 first adders constitute three product-accumulators, wherein each of the product-accumulators includes M / 3 multipliers and M / 3-1 first adders.

13. The device according to claim 1 or 2, characterized in that The two-dimensional convolution kernel is a two-dimensional matrix component of an N-dimensional convolution kernel, where N is an integer greater than 2.

14. A data processing method, characterized in that: include: Obtaining an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the first initial parameter is K, K is a non-zero number, and the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; Obtaining an initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with an initial convolution kernel of the Lth layer; Expand each parameter of the initial convolution kernel of the L+1th layer by K times; The equivalent convolution kernel of the L+1th layer is determined according to the initial convolution kernel of the L+1th layer after the expansion process.

15. The method according to claim 14, characterized in that The equivalent convolution kernel of the Lth layer includes M second parameters and one first parameter, the M second parameters respectively correspond to M second eigenvalues ​​of the feature map, the first parameter corresponds to the first eigenvalue of the feature map, and the first parameter is 1; The step of performing convolution processing on the feature map of the Lth layer includes: Performing a multiplication-accumulation operation on the M second parameters and the M second eigenvalues ​​to obtain a multiplication-accumulation result; An addition operation is performed on the multiplication-accumulation result and the first eigenvalue.

16. The method according to claim 14, characterized in that The parameters of the L-th layer equivalent convolution kernel are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the L-th layer and the first initial parameters in the initial convolution kernel of the L-th layer, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer reduced by m times and the first initial parameter is not greater than the first threshold.

17. The method according to claim 15, characterized in that The adding operation on the multiplication-accumulation result and the first eigenvalue comprises: Reduce the first eigenvalue by m times; An addition operation is performed on the multiplication-accumulation result and the reduced first eigenvalue.

18. The method according to claim 14, characterized in that The parameters of the L-th layer equivalent convolution kernel are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the L-th layer and the first initial parameters in the initial convolution kernel of the L-th layer, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is enlarged n times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer enlarged n times and the first initial parameter is not less than the second threshold.

19. The method according to claim 15, characterized in that The adding operation on the multiplication-accumulation result and the first eigenvalue comprises: Enlarging the first eigenvalue by n times; An addition operation is performed on the multiplication-accumulation result and the enlarged first eigenvalue.

20. A data processing device, characterized in that: include: A processing module, configured to obtain an equivalent convolution kernel of the Lth layer according to the initial convolution kernel of the Lth layer of the neural network, wherein the parameters of the equivalent convolution kernel of the Lth layer are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the Lth layer and the first initial parameter in the initial convolution kernel of the Lth layer, the value of the first initial parameter is K, K is a non-zero number, and the equivalent convolution kernel of the Lth layer is used to perform convolution processing on the feature map of the Lth layer; An acquisition module, used for acquiring an initial convolution kernel of the L+1th layer in the L+1th layer of the neural network that has a mapping relationship with an initial convolution kernel of the Lth layer; An expansion module, used to expand each parameter in the initial convolution kernel of the L+1th layer by K times; A determination module is used to determine the equivalent convolution kernel of the L+1th layer according to the initial convolution kernel of the L+1th layer after the expansion process.

21. The device according to claim 20, characterized in that The equivalent convolution kernel of the Lth layer includes M second parameters and one first parameter, the M second parameters respectively correspond to M second eigenvalues ​​of the feature map, the first parameter corresponds to the first eigenvalue of the feature map, and the first parameter is 1; The step of performing convolution processing on the feature map of the Lth layer includes: Performing a multiplication-accumulation operation on the M second parameters and the M second eigenvalues ​​to obtain a multiplication-accumulation result; An addition operation is performed on the multiplication-accumulation result and the first eigenvalue.

22. The device according to claim 20, characterized in that The parameters of the L-th layer equivalent convolution kernel are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the L-th layer and the first initial parameters in the initial convolution kernel of the L-th layer, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is greater than a first threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is reduced by m times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer reduced by m times and the first initial parameter is not greater than the first threshold.

23. The device according to claim 21, characterized in that The adding operation on the multiplication-accumulation result and the first eigenvalue comprises: Reduce the first eigenvalue by m times; An addition operation is performed on the multiplication-accumulation result and the reduced first eigenvalue.

24. The device according to claim 20, characterized in that The parameters of the L-th layer equivalent convolution kernel are obtained based on the quotient of the corresponding parameters of the initial convolution kernel of the L-th layer and the first initial parameters in the initial convolution kernel of the L-th layer, including: When the quotient of at least one parameter of the initial convolution kernel of the Lth layer and the first initial parameter is less than a second threshold, the corresponding parameter of the initial convolution kernel of the Lth layer is enlarged n times, wherein the quotient of any parameter of the initial convolution kernel of the Lth layer enlarged n times and the first initial parameter is not less than the second threshold.

25. The device according to claim 21, characterized in that The adding operation on the multiplication-accumulation result and the first eigenvalue comprises: Enlarging the first eigenvalue by n times; An addition operation is performed on the multiplication-accumulation result and the enlarged first eigenvalue.

26. A computer-readable storage medium, characterized in that: The method comprises instructions, and when the instructions are executed on a processor, the processor performs the method according to any one of claims 14 to 19.

27. A data processing device, characterized in that: It comprises a programmable device and a memory, wherein the memory is used to store a configuration file required for the operation of the programmable device, and the programmable device is used to read the configuration file from the memory to execute any method as claimed in claims 14 to 19.

28. The device according to claim 27, characterized in that The programmable device includes a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).