Convolutional Neural Network Computing Devices, Methods, and Mobile Terminals

Abstract

This application provides a convolutional neural network (CNN) computing device, method, and mobile terminal. The device includes: a determining module, used to determine multiple target reconfigurable computing units from multiple reconfigurable computing units based on the structure of the CNN model to be processed; a configuring module, used to configure the computation types of each target reconfigurable computing unit and the connection relationships between the multiple target reconfigurable computing units according to the structure of the CNN model; a broadcasting module, used to broadcast data from the CNN model to the corresponding target reconfigurable computing units for computation according to their respective computation types, and to obtain the computation results; and a computing module, used to process the multiple computation results according to the connection relationships to obtain the model output of the CNN model. The solution of this application reduces the computation time consumed by the CNN model and improves the computational efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a convolutional neural network computing device, method, and mobile terminal. Background Technology

[0002] With the development of artificial intelligence technology, the application of convolutional neural network models is becoming more and more widespread.

[0003] Convolutional neural network (CNN) models typically include multiple processing layers, such as convolutional layers, pooling layers, and fully connected layers, each requiring corresponding computational processing. The numerous parameters and computational demands of each layer in a CNN model, involving substantial data transmission and manipulation, result in prolonged computation time and relatively low efficiency.

[0004] Therefore, there is an urgent need to provide a convolutional neural network operation scheme to reduce the time consumed by the convolutional neural network model and improve the efficiency of operation. Summary of the Invention

[0005] This application provides a convolutional neural network computing device, method, and mobile terminal to reduce the time consumed by convolutional neural network model computation and improve computation efficiency.

[0006] In a first aspect, this application provides a convolutional neural network computing device, wherein a plurality of reconfigurable computing units are configured in the convolutional neural network computing device, and the device includes:

[0007] The determining module is used to determine multiple target reconfigurable computing units from the multiple reconfigurable computing units based on the structure of the convolutional neural network model to be processed;

[0008] The configuration module is used to configure the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units according to the structure of the convolutional neural network model. The computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing.

[0009] The broadcast module is used to broadcast the data in the convolutional neural network model to the corresponding target reconfigurable computing units for calculation according to the calculation type of each of the multiple target reconfigurable computing units, so as to obtain the calculation results of each of the multiple target reconfigurable computing units. In this case, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units.

[0010] The calculation module is used to process the calculation results of each of the multiple target reconfigurable calculation units according to the connection relationship between the multiple target reconfigurable calculation units, so as to obtain the model output result of the convolutional neural network model.

[0011] In one possible implementation, the reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein:

[0012] The input interface includes a first input interface, a second input interface, a third input interface, a fourth input interface, a fifth input interface, and a sixth input interface;

[0013] The output interface includes a first output interface and a second output interface;

[0014] The computational subunit includes a comparator, a multiplier, an adder, a first gating unit, a second gating unit, and a third gating unit;

[0015] The input terminal of the comparator is connected to the first input interface and the second input interface, and the output terminal of the comparator is connected to the first output interface;

[0016] The input terminal of the multiplier is connected to the third input interface and the fourth input interface, and the output terminal of the multiplier is connected to the input terminal of the first selector and the first output interface;

[0017] The input terminal of the first selector is also connected to the fifth input interface, the input terminal of the second selector is connected to the sixth input interface, and the output terminals of the first selector and the second selector are both connected to the input terminal of the adder.

[0018] The output of the adder is connected to the input of the third selector;

[0019] The output of the third selector is connected to the input of the second selector and the second output interface.

[0020] In one possible implementation, the calculation type is a comparison process; wherein:

[0021] The first input interface is used to acquire first input data and send the first input data to the comparator;

[0022] The second input interface is used to acquire second input data and send the second input data to the comparator;

[0023] The comparator is used to compare the first input data and the second input data to determine the larger value between the first input data and the second input data.

[0024] The first output interface is used to output the larger value between the first input data and the second input data.

[0025] In one possible implementation, the calculation type is a multiplication operation or a multiply-accumulate operation; wherein:

[0026] The third input interface is used to acquire third input data and send the third input data to the multiplier;

[0027] The fourth input interface is used to acquire fourth input data and send the fourth input data to the multiplier;

[0028] The multiplier is used to perform multiplication on the third input data and the fourth input data to obtain the product of the third input data and the fourth input data.

[0029] In one possible implementation,

[0030] If the calculation type is the multiplication operation, then:

[0031] The multiplier is also used to send the product to the first output interface;

[0032] The first output interface is used to output the product;

[0033] If the calculation type is the multiply-accumulate process, then:

[0034] The multiplier is also used to send the product to the first gate;

[0035] The first gate is used to select which product to send to the adder;

[0036] The third selector is used to select the first output data selected by the third selector to be input into the second selector;

[0037] The second selector is used to select which output data to send to the adder;

[0038] The adder is used to add the first output data and the product to obtain the sum of the first output data and the product;

[0039] The second output interface is used to output the sum of the first output data and the product.

[0040] In one possible implementation, the calculation type is addition; wherein:

[0041] The fifth input interface is used to acquire fifth input data and send the fifth input data to the first selector;

[0042] The first gate is used to select which input data to send to the adder;

[0043] The sixth input interface is used to acquire a sixth input value and send the sixth input value to the second selector;

[0044] The second gate is used to select which input data to send to the adder;

[0045] The adder is used to perform addition on the fifth input data and the sixth input data to obtain the sum of the fifth input data and the sixth input data;

[0046] The second output interface is used to output the sum of the fifth input data and the sixth input data.

[0047] In one possible implementation, the calculation type is an accumulation process; wherein:

[0048] The fifth input interface is used to acquire the seventh input data and send the seventh input data to the first selector;

[0049] The first selector is used to select which input data to send to the adder;

[0050] The third selector is used to select the input of the second output value of the third selector to the second selector;

[0051] The second selector is used to select which output value to send to the adder;

[0052] The adder is used to accumulate the second output value and the seventh input data to obtain the accumulated value of the second output value and the seventh input data;

[0053] The second output interface is used to output the sum of the second output value and the seventh input data.

[0054] In one possible implementation, the structure of the convolutional neural network model includes convolutional layers; the determining module is specifically used for:

[0055] Based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from the plurality of reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix. 3The first objective is a reconfigurable computing unit, according to n 2 Arranged in rows and n columns, where n is a positive integer;

[0056] The configuration module is specifically used for:

[0057] Configure n according to the convolution kernel matrix. 3 The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of the n second target reconfigurable computing units is the accumulation process.

[0058] Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

[0059] Secondly, this application provides a convolutional neural network operation method, applied to a convolutional neural network operation device, wherein the convolutional neural network operation device is configured with multiple reconfigurable computing units, and the method includes:

[0060] Based on the structure of the convolutional neural network model to be processed, multiple target reconfigurable computing units are determined from the multiple reconfigurable computing units;

[0061] Based on the structure of the convolutional neural network model, the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units are configured. The computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing.

[0062] According to the calculation type of each of the plurality of target reconfigurable computing units, the data in the convolutional neural network model is broadcast to the corresponding target reconfigurable computing unit for calculation, and the calculation results of each of the plurality of target reconfigurable computing units are obtained. In this case, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units.

[0063] Based on the connection relationships between the multiple target reconfigurable computing units, the calculation results of each of the multiple target reconfigurable computing units are processed to obtain the model output result of the convolutional neural network model.

[0064] In one possible implementation, the reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein:

[0065] The input interface includes a first input interface, a second input interface, a third input interface, a fourth input interface, a fifth input interface, and a sixth input interface;

[0066] The output interface includes a first output interface and a second output interface;

[0067] The computational subunit includes a comparator, a multiplier, an adder, a first gating unit, a second gating unit, and a third gating unit;

[0068] The input terminal of the comparator is connected to the first input interface and the second input interface, and the output terminal of the comparator is connected to the first output interface;

[0069] The input terminal of the multiplier is connected to the third input interface and the fourth input interface, and the output terminal of the multiplier is connected to the input terminal of the first selector and the first output interface;

[0070] The input terminal of the first selector is also connected to the fifth input interface, the input terminal of the second selector is connected to the sixth input interface, and the output terminals of the first selector and the second selector are both connected to the input terminal of the adder.

[0071] The output of the adder is connected to the input of the third selector;

[0072] The output of the third selector is connected to the input of the second selector and the second output interface.

[0073] In one possible implementation, the calculation type is a comparison process; wherein:

[0074] The first input interface is used to acquire first input data and send the first input data to the comparator;

[0075] The second input interface is used to acquire second input data and send the second input data to the comparator;

[0076] The comparator is used to compare the first input data and the second input data to determine the larger value between the first input data and the second input data.

[0077] The first output interface is used to output the larger value between the first input data and the second input data.

[0078] In one possible implementation, the calculation type is a multiplication operation or a multiply-accumulate operation; wherein:

[0079] The third input interface is used to acquire third input data and send the third input data to the multiplier;

[0080] The fourth input interface is used to acquire fourth input data and send the fourth input data to the multiplier;

[0081] The multiplier is used to perform multiplication on the third input data and the fourth input data to obtain the product of the third input data and the fourth input data.

[0082] In one possible implementation,

[0083] If the calculation type is the multiplication operation, then:

[0084] The multiplier is also used to send the product to the first output interface;

[0085] The first output interface is used to output the product;

[0086] If the calculation type is the multiply-accumulate process, then:

[0087] The multiplier is also used to send the product to the first gate;

[0088] The first gate is used to select which product to send to the adder;

[0089] The third selector is used to select the first output data selected by the third selector to be input into the second selector;

[0090] The second selector is used to select which output data to send to the adder;

[0091] The adder is used to add the first output data and the product to obtain the sum of the first output data and the product;

[0092] The second output interface is used to output the sum of the first output data and the product.

[0093] In one possible implementation, the calculation type is addition; wherein:

[0094] The fifth input interface is used to acquire fifth input data and send the fifth input data to the first selector;

[0095] The first gate is used to select which input data to send to the adder;

[0096] The sixth input interface is used to acquire a sixth input value and send the sixth input value to the second selector;

[0097] The second gate is used to select which input data to send to the adder;

[0098] The adder is used to perform addition on the fifth input data and the sixth input data to obtain the sum of the fifth input data and the sixth input data;

[0099] The second output interface is used to output the sum of the fifth input data and the sixth input data.

[0100] In one possible implementation, the calculation type is an accumulation process; wherein:

[0101] The fifth input interface is used to acquire the seventh input data and send the seventh input data to the first selector;

[0102] The first selector is used to select which input data to send to the adder;

[0103] The third selector is used to select the input of the second output value of the third selector to the second selector;

[0104] The second selector is used to select which output value to send to the adder;

[0105] The adder is used to accumulate the second output value and the seventh input data to obtain the accumulated value of the second output value and the seventh input data;

[0106] The second output interface is used to output the sum of the second output value and the seventh input data.

[0107] In one possible implementation, the structure of the convolutional neural network model includes convolutional layers; determining a plurality of target reconfigurable computational units from the plurality of reconfigurable computational units based on the structure of the convolutional neural network model to be processed includes:

[0108] Based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from the plurality of reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix. 3 The first objective is a reconfigurable computing unit, according to n 2 Arranged in rows and n columns, where n is a positive integer;

[0109] The step of configuring the computation types of the plurality of target reconfigurable computation units and the connection relationships between the plurality of target reconfigurable computation units according to the structure of the convolutional neural network model includes:

[0110] Configure n according to the convolution kernel matrix. 3The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of the n second target reconfigurable computing units is the accumulation process.

[0111] Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

[0112] Thirdly, this application provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the convolutional neural network operation method as described in any of the second aspects.

[0113] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the convolutional neural network operation method as described in any of the second aspects.

[0114] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the convolutional neural network operation method as described in any of the second aspects.

[0115] The convolutional neural network (CNN) computing device, method, and mobile terminal provided in this application embodiment are as follows: The CNN computing device is configured with multiple reconfigurable computing units. Based on the structure of the CNN model to be processed, the CNN computing device determines multiple target reconfigurable computing units from among the multiple reconfigurable computing units. Then, based on the structure of the CNN model, it configures the computation type of each target reconfigurable computing unit and the connection relationships between the multiple target reconfigurable computing units. According to the computation type of each target reconfigurable computing unit, the data in the CNN model is broadcast to the corresponding target reconfigurable computing unit for computation, obtaining the computation results of each target reconfigurable computing unit. In this case, at least one piece of data in the CNN model is broadcast to at least two target reconfigurable computing units. Based on the connection relationships between the multiple target reconfigurable computing units, the computation results of each target reconfigurable computing unit are processed to obtain the model output result of the CNN model. Because reconfigurable computing units can be configured for different computation types, including comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing, the operation of convolutional neural network models can be easily ported to mobile terminals for processing. During the operation of the convolutional neural network model, data is transmitted through broadcasting, with at least one piece of data being broadcast to at least two target reconfigurable computing units. This broadcast data transmission method can reduce the time consumed by data transmission between different units, greatly improve the data transmission rate and the parallelism of the operation, reduce the time consumed by the convolutional neural network model operation, improve the efficiency of the operation, and reduce clock waste to a certain extent. Attached Figure Description

[0116] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0117] Figure 1 This is a schematic diagram of the structure of the convolutional neural network computing device provided in the embodiments of this application;

[0118] Figure 2 This is a schematic diagram of the reconfigurable computing unit provided in an embodiment of this application;

[0119] Figure 3 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 1 ;

[0120] Figure 4This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 2 ;

[0121] Figure 5 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 3 ;

[0122] Figure 6 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 4 ;

[0123] Figure 7 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 5 ;

[0124] Figure 8 A schematic diagram of the reconfigurable computational unit architecture corresponding to the convolutional layer provided in the embodiments of this application. Figure 1 ;

[0125] Figure 9 A schematic diagram of a reconfigurable computational unit architecture based on convolutional layers provided in the embodiments of this application. Figure 2 ;

[0126] Figure 10 A schematic diagram of a reconfigurable computational unit architecture based on convolutional layers provided in the embodiments of this application. Figure 3 ;

[0127] Figure 11 A flowchart illustrating the convolutional neural network operation method provided in this application embodiment;

[0128] Figure 12 This is a schematic diagram of the physical structure of a mobile terminal provided in an embodiment of this application. Detailed Implementation

[0129] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0130] Artificial intelligence pattern recognition technology is a well-known technology that involves the computation of convolutional neural network (CNN) models. CNN models maintain a unique scientific and technological advantage in image recognition due to their special network structure for local data extraction and weight data sharing. The network layout of CNN models closely resembles actual biological neural networks, and the weight sharing method greatly reduces the difficulty and complexity of implementing neural network-based image recognition systems.

[0131] With the development of mobile IoT and the popularization of portable machine learning devices, due to limitations in the security, real-time performance, and bandwidth of convolutional neural network models, it is imperative to transfer some inference data to mobile terminals.

[0132] The advantage of mobile terminals lies in reducing communication and request response time with the cloud, and increasing data security and privacy. However, convolutional neural network models involve the computation of multiple hidden layers, including convolutional layers, pooling layers, and fully connected layers. Different neural networks also have the problem of different convolutional kernel sizes, resulting in poor transferability.

[0133] Convolutional neural network (CNN) models typically include multiple processing layers, such as convolutional layers, pooling layers, and fully connected layers, each requiring specific computational processing. When deploying CNN models on mobile devices, this involves configuring appropriate computational units for the CNN model, transferring data from the CNN model to these units, and then outputting the results. Because CNN models have numerous parameters across layers, resulting in a large computational load and extensive data transfer and processing, the computational process is time-consuming and inefficient.

[0134] Based on this, embodiments of this application provide a convolutional neural network operation method to improve the efficiency of convolutional neural network model operations. The solutions of this application embodiment will be described below with reference to the accompanying drawings.

[0135] Figure 1 This is a schematic diagram of the structure of the convolutional neural network computing device provided in the embodiments of this application. The convolutional neural network computing device is configured with multiple reconfigurable computing units, such as... Figure 1 As shown, the convolutional neural network computing device includes:

[0136] The determination module 11 is used to determine multiple target reconfigurable computational units from multiple reconfigurable computational units based on the structure of the convolutional neural network model to be processed.

[0137] A convolutional neural network (CNN) model can include multiple different layers, such as convolutional layers, pooling layers, fully connected layers, sampling layers, etc. The number of different layers may be equal or unequal, and these layers have corresponding connections. These different layers and the connections between them constitute the structure of the CNN model.

[0138] The execution entity in this application embodiment is a convolutional neural network (CNN) computing device. For example, a CNN model from a mobile terminal can be embedded within the CNN computing device for computation. Multiple reconfigurable computing units are pre-configured within the CNN computing device. The reconfigurable computing units in this application embodiment have the same structure, but can be configured for different computation types according to actual needs to adapt to the computation of different CNN models and different layers within the CNN model.

[0139] For the convolutional neural network model to be processed, after determining the structure of the convolutional neural network model, the convolutional neural network computing device determines multiple target reconfigurable computing units among the multiple reconfigurable computing units configured in the mobile terminal according to the structure of the convolutional neural network model. These multiple target reconfigurable computing units are the reconfigurable computing units used for subsequent operations of the convolutional neural network model.

[0140] A convolutional neural network model can be divided into multiple layers. For each layer, the target reconfigurable computational units required for that layer can be determined based on the specific computational process of that layer. Then, the target reconfigurable computational units required by the different layers of the convolutional neural network model together constitute the multiple target reconfigurable computational units of the convolutional neural network model.

[0141] Optionally, if there are two layers in the convolutional neural network model whose computations do not interfere with each other, or have a temporal order, then the target reconfigurable computational units required by these two layers can be reused. That is, some target reconfigurable computational units can be used for the computation of the other layer after completing the computation of one layer.

[0142] For example, if a first convolutional layer and a second convolutional layer are connected, and the output of the first convolutional layer is the input of the second convolutional layer, then the operations of the first convolutional layer and the second convolutional layer have a sequential order. After the first convolutional layer determines the target reconfigurable computing unit and performs the calculation, some or all of the target reconfigurable computing units required by the first convolutional layer can be used for the operation of the second convolutional layer to save the computing resources of the target reconfigurable computing unit.

[0143] Configuration module 12 is used to configure the computation type of each of the multiple target reconfigurable computation units and the connection relationship between the multiple target reconfigurable computation units according to the structure of the convolutional neural network model. The computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing.

[0144] After identifying multiple target reconfigurable computing units, the mobile terminal configures the computing types of each target reconfigurable computing unit according to the structure of the convolutional neural network model. As mentioned above, the multiple reconfigurable computing units have the same structure, but can be configured with different computing types. These types of techniques may include, for example, comparison processing, addition processing, multiplication processing, accumulation processing, multiply-accumulate processing, etc.

[0145] Comparison processing compares two input data to output the larger or smaller value; addition processing adds two input data; multiplication processing multiplies two input data; accumulation processing adds the current value to a new input value; multiply-accumulate processing multiplies two input data and adds the result to the current value.

[0146] After determining the multiple target reconfigurable computational units required for the convolutional neural network (CNN) model, the computational type of each target reconfigurable computational unit can be configured according to the structure of the CNN model. That is, each target reconfigurable computational unit can be configured to perform a specific computational operation. For example, if the CNN model includes convolutional layers, which typically involve multiplication and addition, then the corresponding target reconfigurable computational units can be configured with multiplication, addition, or multiply-accumulate operations. Similarly, if the CNN model includes normalization layers, which also typically involve multiplication, then the corresponding target reconfigurable computational units can also be configured with multiplication. The specific configuration method can be set according to the structure of the CNN model, and this embodiment does not limit this.

[0147] Broadcast module 13 is used to broadcast the data in the convolutional neural network model to the corresponding target reconfigurable computing units for calculation according to the calculation type of each target reconfigurable computing unit, so as to obtain the calculation results of each target reconfigurable computing unit. In this case, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units.

[0148] After configuring the corresponding computation types for multiple target reconfigurable computational units, it is necessary to input the corresponding data for each target reconfigurable computational unit to perform the computation. In the convolutional neural network model, different layers have corresponding data. For any given layer, the computation process is performed based on the specific operations of that layer and the corresponding data to obtain the computation results of each of the multiple target reconfigurable computational units.

[0149] In this embodiment, the data input process is performed in a broadcast manner. The mobile terminal broadcasts the data from the convolutional neural network model to the corresponding target reconfigurable computing units for computation. That is, the mobile terminal directly inputs the data from the convolutional neural network model into the registers of the corresponding target reconfigurable computing units in a broadcast manner. Optionally, during the data transmission in a broadcast manner, all target reconfigurable computing units can receive each piece of data broadcast by the mobile terminal. Therefore, when broadcasting each piece of data, the mobile terminal can carry an indication to indicate which target reconfigurable computing units the data needs to be transmitted to, so that each piece of data is transmitted to the corresponding target reconfigurable computing unit.

[0150] It should be noted that in a convolutional neural network model, at least one piece of data is broadcast to at least two target reconfigurable computational units. For example, regarding the convolution kernel in a convolutional neural network model, the elements of the convolution kernel are simultaneously broadcast to multiple target reconfigurable computational units. This broadcasting method can improve the data transmission rate and reduce the time consumed by data transmission.

[0151] The calculation module 14 is used to process the calculation results of each of the multiple target reconfigurable calculation units according to the connection relationship between the multiple target reconfigurable calculation units, so as to obtain the model output result of the convolutional neural network model.

[0152] After the mobile terminal obtains the calculation results by calculating the data of each of the multiple target reconfigurable computing units, it processes the calculation results of each of the multiple target reconfigurable computing units according to the connection relationship between the multiple target reconfigurable computing units.

[0153] The connection relationships between the multiple target reconfigurable computing units referred to in this application embodiment are determined by the structure of the convolutional neural network model. In the convolutional neural network model, for each layer, the specific computation process within that layer, as well as the input and output of that computation process, are implemented through the connection relationships between the multiple target reconfigurable computing units. In other words, the data operations, data transmission, and flow processes involved in the convolutional neural network model constitute the connection relationships between the multiple target reconfigurable computing units. The mobile terminal processes the computation results of each of the multiple target reconfigurable computing units according to these connection relationships to obtain the model output of the convolutional neural network model.

[0154] Based on any of the above embodiments, the solutions of the embodiments of this application will be described in detail below.

[0155] First, combine Figure 2 The structure of the reconfigurable computing unit in the embodiments of this application is described.

[0156] Figure 2 This is a schematic diagram of the reconfigurable computing unit provided in an embodiment of this application, as shown below. Figure 2 As shown, the reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein:

[0157] The input interfaces include a first input interface A, a second input interface B, a third input interface C, a fourth input interface D, a fifth input interface E, and a sixth input interface F;

[0158] The output interfaces include a first output interface X and a second output interface Y;

[0159] The computation subunit includes a comparator 21, a multiplier 22, an adder 23, a first selector 24, a second selector 25, and a third selector 26;

[0160] The input terminal of comparator 21 is connected to the first input interface A and the second input interface B, and the output terminal of comparator 21 is connected to the first output interface X;

[0161] The input terminals of multiplier 22 are connected to the third input interface C and the fourth input interface D, and the output terminals of multiplier 22 are connected to the input terminals of the first selector 24 and the first output interface X.

[0162] The input of the first selector 24 is also connected to the fifth input interface E, the input of the second selector 25 is connected to the sixth input interface F, and the outputs of the first selector 24 and the second selector 25 are both connected to the input of the adder 23.

[0163] The output of adder 23 is connected to the input of third selector 26;

[0164] The output of the third selector 26 is connected to the input of the second selector 25 and the second output interface Y.

[0165] exist Figure 2 The example illustrates the structure of a reconfigurable computing unit. The following is based on... Figure 2 The example demonstrates the structure of a reconfigurable computing unit and introduces the computing process when the reconfigurable computing unit is configured for different computing types.

[0166] Figure 3 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 1 ,like Figure 3 As shown, the structure of the example reconfigurable computing unit is similar to... Figure 2 same, Figure 3 The reconfigurable computing unit in the configuration is configured for comparison processing.

[0167] The first input interface A is used to acquire the first input data and send the first input data to the comparator 21;

[0168] The second input interface B is used to acquire the second input data and send the second input data to the comparator 21;

[0169] Comparator 21 is used to compare the first input data and the second input data to determine the larger value between the first input data and the second input data;

[0170] The first output interface X is used to output the larger value between the first input data and the second input data.

[0171] exist Figure 3 The example illustrates the data flow when a reconfigurable computing unit is configured for comparison processing. When the reconfigurable computing unit is configured for comparison processing, its input interfaces are a first input interface A and a second input interface B; data is input from the first input interface A and the second input interface B, and no data is input from any other input interfaces. The output interface of the reconfigurable computing unit is a first output interface X, and the second output interface Y does not output any data.

[0172] The data input from the first input interface A is the first input data. After being input from the first input interface A, the first input data is transmitted to comparator 21 via the first input interface A. The data input from the second input interface B is the second input data. After being input from the second input interface B, the second input data is transmitted to comparator 21 via the second input interface B. Comparator 21 is used to compare the two input data and then output the larger value between the first and second input data. Taking the first input data as 'a' and the second input data as 'b' as an example, if 'a' is greater than or equal to 'b', comparator 21 outputs 'a'; if 'a' is less than 'b', comparator 21 outputs 'b'. The data output by comparator 21 is transmitted to the first output interface X, and the first output interface X outputs the larger value between the first and second input data.

[0173] Figure 4 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 2 ,like Figure 4 As shown, the structure of the example reconfigurable computing unit is similar to... Figure 2 same, Figure 4 The reconfigurable computing unit in the process is configured for multiplication.

[0174] The third input interface C is used to acquire the third input data and send the third input data to the multiplier 22;

[0175] The fourth input interface D is used to acquire the fourth input data and send the fourth input data to the multiplier 22;

[0176] Multiplier 22 is used to multiply the third input data and the fourth input data to obtain the product of the third input data and the fourth input data; and sends the product to the first output interface X;

[0177] The first output interface X is used to output the product.

[0178] exist Figure 4 The example illustrates the data flow when the reconfigurable computing unit is configured for multiplication. When the reconfigurable computing unit is configured for multiplication, its input interfaces are the third input interface C and the fourth input interface D; data is input from the third input interface C and the fourth input interface D, while other input interfaces do not input any data. The output interface of the reconfigurable computing unit is the first output interface X, and the second output interface Y does not output any data.

[0179] The data input from the third input interface C is the third input data. After being input from the third input interface C, the third input data is transmitted to multiplier 22 via the third input interface C. The data input from the fourth input interface D is the fourth input data. After being input from the fourth input interface D, the fourth input data is transmitted to multiplier 22 via the fourth input interface D. Multiplier 22 is used to multiply the two input data and then output the product of the third input data and the fourth input data. Taking the third input data as c and the fourth input data as d as an example, multiplier 22 multiplies c and d to obtain the product c*d, and outputs the product c*d. The product c*d output by multiplier 22 is transmitted to the first output interface X, and the first output interface X outputs the product c*d.

[0180] Figure 5 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 3 ,like Figure 5 As shown, the structure of the example reconfigurable computing unit is similar to... Figure 2 same, Figure 5 The reconfigurable computing unit in the process is configured for multiply-accumulate processing.

[0181] The third input interface C is used to acquire the third input data and send the third input data to the multiplier 22;

[0182] The fourth input interface D is used to acquire the fourth input data and send the fourth input data to the multiplier 22;

[0183] Multiplier 22 is used to multiply the third input data and the fourth input data to obtain the product of the third input data and the fourth input data; and sends the product to the first selector 24.

[0184] The first selector 24 is used to select which product to send to the adder;

[0185] The third selector 26 is used to select the first output data selected by the third selector 26 to be input to the second selector 25;

[0186] The second selector 25 is used to select which first output data is sent to adder 23;

[0187] Adder 23 is used to add the first output data and the product to obtain the sum of the first output data and the product;

[0188] The second output interface Y is used to output the first output data and the sum of the products.

[0189] exist Figure 5The example illustrates the data flow when the reconfigurable computing unit is configured for multiply-accumulate processing. When the reconfigurable computing unit is configured for multiply-accumulate processing, its input interfaces are the third input interface C and the fourth input interface D; data is input from the third input interface C and the fourth input interface D, while other input interfaces do not input any data. The output interface of the reconfigurable computing unit is the second output interface Y, and the first output interface X does not output any data.

[0190] The data input from the third input interface C is the third input data. After being input from the third input interface C, the third input data is transmitted to multiplier 22 via the third input interface C. The data input from the fourth input interface D is the fourth input data. After being input from the fourth input interface D, the fourth input data is transmitted to multiplier 22 via the fourth input interface D. Multiplier 22 is used to multiply the two input data and then output the product of the third input data and the fourth input data. Taking the third input data as c and the fourth input data as d as an example, multiplier 22 multiplies c and d to obtain the product c*d, and outputs the product c*d. The product c*d output by multiplier 22 is transmitted to the first selector 24. The first selector 24 selects the signal of product c*d, that is, it outputs product c*d to adder 23.

[0191] Before adder 23 receives the product c*d, its output data is the first output data x. This first output data x is output to the third selector 26, which selects to input the first output data x to the second selector 25. Then, the second selector 25 selects to send the first output data x to adder 23. At this point, the inputs of adder 23 include the product c*d and the first output data x. Adder 23 adds the first output data x and the product c*d to obtain the sum of the first output data x and the product c*d, i.e., x + c*d, and sends this sum to the second output interface Y. The second output interface Y outputs the sum of the first output data and the product, i.e., x + c*d.

[0192] Figure 6 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 4 ,like Figure 6 As shown, the structure of the example reconfigurable computing unit is similar to... Figure 2 same, Figure 6 The reconfigurable computing unit in the process is configured for addition.

[0193] The fifth input interface E is used to acquire the fifth input data and send the fifth input data to the first selector;

[0194] The first selector 24 is used to select which fifth input data is sent to the adder 23;

[0195] The sixth input interface F is used to acquire the sixth input value and send the sixth input value to the second selector 25;

[0196] The second selector is used to select which sixth input data is sent to adder 23;

[0197] Adder 23 is used to perform addition on the fifth input data and the sixth input data to obtain the sum of the fifth input data and the sixth input data;

[0198] The second output interface Y is used to output the sum of the fifth and sixth input data.

[0199] exist Figure 6 The example illustrates the data flow when the reconfigurable computing unit is configured for addition processing. When the reconfigurable computing unit is configured for addition processing, its input interfaces are the fifth input interface E and the sixth input interface F; data is input from the fifth input interface E and the sixth input interface F, while other input interfaces do not input any data. The output interface of the reconfigurable computing unit is the second output interface Y, and the first output interface X does not output any data.

[0200] The data input from the fifth input interface E is the fifth input data. After being input from the fifth input interface E, the fifth input data is transmitted from the fifth input interface E to the first selector 24. The first selector 24 selects to send the fifth input data to the adder 23. The data input from the sixth input interface F is the sixth input data. After being input from the sixth input interface F, the sixth input data is transmitted from the sixth input interface F to the second selector 25. The second selector 25 selects to send the sixth input data to the adder 23. The adder 23 is used to add the two input data and then output the sum of the fifth input data and the sixth input data. Taking the fifth input data as e and the sixth input data as f as an example, the adder 23 adds e and f to obtain the sum of e and f, e+f, and outputs the sum e+f. The sum e+f output by the adder 23 is transmitted to the second output interface Y, and the second output interface Y outputs the sum e+f.

[0201] Figure 7 This application provides a schematic diagram of the reconfigurable computing unit processing in an embodiment. Figure 5 ,like Figure 7 As shown, the structure of the example reconfigurable computing unit is similar to... Figure 2 same, Figure 7 The reconfigurable computing unit in the process is configured with an accumulation processing type.

[0202] The fifth input interface E is used to acquire the seventh input data and send the seventh input data to the first selector 24;

[0203] The first selector 24 is used to select whether the seventh input data is sent to the adder 23;

[0204] The third selector 26 is used to select the second output value output by the third selector to be input to the second selector 25;

[0205] The second selector 25 is used to select which output value to send to the adder 23;

[0206] Adder 23 is used to accumulate the second output value and the seventh input data to obtain the accumulated value of the second output value and the seventh input data;

[0207] The second output interface Y is used to output the second output value and the accumulated value of the seventh input data.

[0208] exist Figure 7 The example illustrates the data flow when a reconfigurable computing unit is configured for accumulation processing. When the reconfigurable computing unit is configured for accumulation processing, its input interface is the fifth input interface E, meaning data is input from the fifth input interface E; other input interfaces do not input any data. The output interface of the reconfigurable computing unit is the second output interface Y, and the first output interface X does not output any data.

[0209] The data input from the fifth input interface E is the seventh input data g. After the seventh input data g is input from the fifth input interface E, it is transmitted to the first selector 24. The first selector 24 selects to send the seventh input data g to the adder 23. The third selector 26 is used to select the second output value h output by the third selector to be input to the second selector 25. The second selector 25 selects to send the second output value h to the adder 23. The adder 23 adds the second output value h and the seventh input data g to obtain the accumulated value h+g. The second output interface Y outputs the second output value and the accumulated value h+g of the seventh input data.

[0210] In one possible implementation, the structure of the convolutional neural network model includes convolutional layers. The specific computation scheme is described below based on the reconfigurable computing unit described in the above embodiments.

[0211] First, based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from multiple reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix.3 The first objective is a reconfigurable computing unit, according to n 2 Arranged in rows and n columns.

[0212] Taking a 3x3 matrix as an example, where n=3, the convolution kernel matrix is:

[0213]

[0214] Based on the convolution kernel matrix B mentioned above, 27 first-objective reconfigurable computational units and 3 second-objective reconfigurable computational units can be identified.

[0215] Then, based on the convolution kernel matrix, configure n 3 The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of each of the n second target reconfigurable computing units is accumulation.

[0216] Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

[0217] Figure 8 A schematic diagram of the reconfigurable computational unit architecture corresponding to the convolutional layer provided in the embodiments of this application. Figure 1 ,like Figure 8 As shown, an example of a first-target reconfigurable computational unit is set up with n=3, including n columns n 2 Okay, the first target reconfigurable computational unit is configured for multiplication processing.

[0218] The dataset matrix A is a 5x5 matrix, as shown below:

[0219]

[0220] First, the data is directly input into the register of each first target reconfigurable computing unit in a broadcast manner according to a certain arrangement. The horizontal input is each row of the convolution kernel matrix B, and the data can only be passed horizontally; the diagonal and vertical input is each row of the dataset matrix A, which is divided with a sliding window of 3 and a stride of 1, and the data can only be passed diagonally and vertically.

[0221] like Figure 8 As shown, at the start of the first clock cycle, the dataset matrices of the first, second, and third columns are input data. After multiplication, the data is fed into the accumulator (which is the second target reconfigurable computing unit corresponding to each of the first, second, and third columns) for accumulation before being output. This process continues, and all the data can be calculated after three clock cycles.

[0222] The data flow structure will be analyzed in detail below.

[0223] The first objective is to arrange the reconfigurable computing units as follows: Figure 8 As shown, the convolution kernel matrix B is divided into columns. The values ​​in each column are input sequentially from top to bottom into the first target reconfigurable computational unit. For example, the value 0 in the first row and first column of the convolution kernel matrix B is input into the first target reconfigurable computational unit in the first row; the value 1 in the second row and first column is input into the second target reconfigurable computational unit in the second row; the value 2 in the third row and first column is input into the third target reconfigurable computational unit in the third row, and so on. Here, the data values ​​of the convolution kernel matrix B are fixed in each first target reconfigurable computational unit, and the values ​​of the convolution kernel portion in each row of the first target reconfigurable computational unit are the same. The values ​​of the input matrix are passed along the diagonal direction.

[0224] Next, the arranged data needs to be sequentially passed into all the first target reconfigurable computing units, such as... Figure 8 As shown, the values ​​12, 8, 11, 5, 2, 7, 10, 4, 12, 3, 6, 8, 9, 4, and 13 are all simultaneously input into all the first target reconfigurable computing units for multiplication. It can be seen that some values ​​are input into multiple first target reconfigurable computing units for multiplication. For example, the value 8 in the second row input port is input into the first target reconfigurable computing unit with a convolution value of 0 for multiplication, and then into the first target reconfigurable computing unit with a convolution value of 1 for multiplication; it is input into two first target reconfigurable computing units. Similarly, the value 11 in the third row, first column input port is input into the first target reconfigurable computing unit with a convolution value of 0 for multiplication, then into the first target reconfigurable computing unit with a convolution value of 1 for multiplication, and then into the first target reconfigurable computing unit with a convolution value of 2 for multiplication; it is input into three first target reconfigurable computing units.

[0225] Figure 9 A schematic diagram of a reconfigurable computational unit architecture based on convolutional layers provided in the embodiments of this application. Figure 2 ,like Figure 9 As shown, each of the first target reconfigurable computing units in the first column inputs the calculated value into the accumulator (which is the corresponding second target reconfigurable computing unit) for accumulation and output. This value is the value in the first row and first column of the convolution result matrix. At the same time, each of the first target reconfigurable computing units in the second and third columns inputs the calculated value into their respective accumulators for accumulation and output. These two values ​​are the values ​​in the second row and first column of the convolution result matrix and the values ​​in the third row and first column, respectively.

[0226] After the multiplication and accumulation operations are completed and the multiplied and accumulated values ​​are output, the multiplication result value in each first target reconstructable computational unit is cleared, and the accumulated value in all accumulators is also cleared. The value of the convolution kernel used in the previous calculation remains unchanged and is still retained in each first target reconstructable computational unit. However, the value of the dataset matrix A used in the previous calculation is no longer piped through, but is instead re-passed into the new input matrix at the input end of each first target reconstructable computational unit.

[0227] Figure 10 A schematic diagram of a reconfigurable computational unit architecture based on convolutional layers provided in the embodiments of this application. Figure 3 ,like Figure 10 As shown, in the second pass, the values ​​5, 2, 7, 10, 4, 12, 8, 1, 0, 8, 9, 4, 13, 12, and 11 are fed into all the first target reconfigurable computing units and the values ​​in the convolution kernel for multiplication. Then, each of the first target reconfigurable computing units in the first, second, and third columns passes the calculated values ​​into the accumulator for accumulation and output. These three values ​​are the values ​​in the first row and second column, the second row and second column, and the third row and second column of the convolution result matrix, respectively.

[0228] Similarly, after the values ​​of dataset matrix A are passed to each first target reconfigurable computational unit for the third time, the values ​​in the first row and third column, the second row and third column, and the third row and third column of the convolution result matrix can be obtained, such as... Figure 10 As shown on the right.

[0229] After all the input data has been calculated according to the above data transfer rules, a completely correct convolution result will be obtained.

[0230] Analysis of the above computational structure shows that the result calculated by the first target reconfigurable computational unit in the first column is the value in the first row of the convolution result matrix, the result calculated by the second target reconfigurable computational unit in the second column is the value in the second row of the convolution result matrix, and the result calculated by the third target reconfigurable computational unit in the third column is the value in the third row of the convolution result matrix.

[0231] In this neural network acceleration structure based on a broadcast-like structure, the convolution kernel value is fixed in each first target reconfigurable computing unit. As long as the convolution kernel remains unchanged during the convolution operation, the value of the convolution kernel in the first target reconfigurable computing unit can remain fixed. The reuse of the diagonal input value requires a certain transformation of the original input matrix, that is, the elements of each row of the dataset matrix A are divided with a sliding window of 3 and a step size of 1. The divided data groups are input to the data input end of each first target reconfigurable computing unit.

[0232] For example, [12, 5, 10] in the first row of dataset matrix A is input to the first row of the first target reconfigurable computational unit; [8, 2, 4] in the second row of dataset matrix A is input to the second row of the first target reconfigurable computational unit; [11, 7, 12] in the third row of dataset matrix A is input to the first column of the third row of the first target reconfigurable computational unit; [3, 8, 4] in the fourth row of dataset matrix A is input to the second column of the third row of the first target reconfigurable computational unit; and [6, 9, 13] in the fifth row of dataset matrix A is input to the third column of the third row of the first target reconfigurable computational unit. Third, the number of rows in the first target reconfigurable computational unit depends on the number of columns in the convolution kernel matrix multiplied by the number of groups in each row after the dataset matrix A is segmented.

[0233] It can be seen that this structure can greatly save the time of data transmission and retrieval. In summary, the data throughput of this algorithm is theoretically three times that of the traditional algorithm. Although it comes at the cost of a certain area overhead, the time consumption is only one-third of the original, thus achieving the goal of acceleration.

[0234] In summary, the convolutional neural network (CNN) computing device provided in this application embodiment is configured with multiple reconfigurable computing units. Based on the structure of the CNN model to be processed, the CNN computing device determines multiple target reconfigurable computing units from among the multiple reconfigurable computing units. Then, based on the structure of the CNN model, it configures the computation type of each target reconfigurable computing unit and the connection relationships between the multiple target reconfigurable computing units. According to the computation type of each target reconfigurable computing unit, the data in the CNN model is broadcast to the corresponding target reconfigurable computing unit for computation, obtaining the computation results of each target reconfigurable computing unit. In this case, at least one piece of data in the CNN model is broadcast to at least two target reconfigurable computing units. Based on the connection relationships between the multiple target reconfigurable computing units, the computation results of each target reconfigurable computing unit are processed to obtain the model output result of the CNN model. Because reconfigurable computing units can be configured for different computation types, including comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing, the operation of convolutional neural network models can be easily ported to mobile terminals for processing. During the operation of the convolutional neural network model, data is transmitted through broadcasting, with at least one piece of data being broadcast to at least two target reconfigurable computing units. This broadcast data transmission method can reduce the time consumed by data transmission between different units, greatly improve the data transmission rate and the parallelism of the operation, reduce the time consumed by the convolutional neural network model operation, improve the efficiency of the operation, and reduce clock waste to a certain extent.

[0235] The convolutional neural network operation method provided in this application is described below. The convolutional neural network operation method described below can be referred to in correspondence with the convolutional neural network operation device described above.

[0236] Figure 11 This is a flowchart illustrating the convolutional neural network operation method provided in an embodiment of this application, applied to a convolutional neural network operation device. The convolutional neural network operation device is configured with multiple reconfigurable computing units, such as... Figure 11 As shown, the method includes:

[0237] S111, Based on the structure of the convolutional neural network model to be processed, determine a plurality of target reconfigurable computing units from the plurality of reconfigurable computing units;

[0238] S112, Based on the structure of the convolutional neural network model, configure the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units, wherein the computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing;

[0239] S113, according to the calculation type of each of the plurality of target reconfigurable computing units, the data in the convolutional neural network model is broadcast to the corresponding target reconfigurable computing unit for calculation, and the calculation results of each of the plurality of target reconfigurable computing units are obtained, wherein at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units;

[0240] S114, based on the connection relationship between the multiple target reconfigurable computing units, process the calculation results of each of the multiple target reconfigurable computing units to obtain the model output result of the convolutional neural network model.

[0241] In one possible implementation, the reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein:

[0242] The input interface includes a first input interface, a second input interface, a third input interface, a fourth input interface, a fifth input interface, and a sixth input interface;

[0243] The output interface includes a first output interface and a second output interface;

[0244] The computational subunit includes a comparator, a multiplier, an adder, a first gating unit, a second gating unit, and a third gating unit;

[0245] The input terminal of the comparator is connected to the first input interface and the second input interface, and the output terminal of the comparator is connected to the first output interface;

[0246] The input terminal of the multiplier is connected to the third input interface and the fourth input interface, and the output terminal of the multiplier is connected to the input terminal of the first selector and the first output interface;

[0247] The input terminal of the first selector is also connected to the fifth input interface, the input terminal of the second selector is connected to the sixth input interface, and the output terminals of the first selector and the second selector are both connected to the input terminal of the adder.

[0248] The output of the adder is connected to the input of the third selector;

[0249] The output of the third selector is connected to the input of the second selector and the second output interface.

[0250] In one possible implementation, the calculation type is a comparison process; wherein:

[0251] The first input interface is used to acquire first input data and send the first input data to the comparator;

[0252] The second input interface is used to acquire second input data and send the second input data to the comparator;

[0253] The comparator is used to compare the first input data and the second input data to determine the larger value between the first input data and the second input data.

[0254] The first output interface is used to output the larger value between the first input data and the second input data.

[0255] In one possible implementation, the calculation type is a multiplication operation or a multiply-accumulate operation; wherein:

[0256] The third input interface is used to acquire third input data and send the third input data to the multiplier;

[0257] The fourth input interface is used to acquire fourth input data and send the fourth input data to the multiplier;

[0258] The multiplier is used to perform multiplication on the third input data and the fourth input data to obtain the product of the third input data and the fourth input data.

[0259] In one possible implementation,

[0260] If the calculation type is the multiplication operation, then:

[0261] The multiplier is also used to send the product to the first output interface;

[0262] The first output interface is used to output the product;

[0263] If the calculation type is the multiply-accumulate process, then:

[0264] The multiplier is also used to send the product to the first gate;

[0265] The first gate is used to select which product to send to the adder;

[0266] The third selector is used to select the first output data selected by the third selector to be input into the second selector;

[0267] The second selector is used to select which output data to send to the adder;

[0268] The adder is used to add the first output data and the product to obtain the sum of the first output data and the product;

[0269] The second output interface is used to output the sum of the first output data and the product.

[0270] In one possible implementation, the calculation type is addition; wherein:

[0271] The fifth input interface is used to acquire fifth input data and send the fifth input data to the first selector;

[0272] The first gate is used to select which input data to send to the adder;

[0273] The sixth input interface is used to acquire a sixth input value and send the sixth input value to the second selector;

[0274] The second gate is used to select which input data to send to the adder;

[0275] The adder is used to perform addition on the fifth input data and the sixth input data to obtain the sum of the fifth input data and the sixth input data;

[0276] The second output interface is used to output the sum of the fifth input data and the sixth input data.

[0277] In one possible implementation, the calculation type is an accumulation process; wherein:

[0278] The fifth input interface is used to acquire the seventh input data and send the seventh input data to the first selector;

[0279] The first selector is used to select which input data to send to the adder;

[0280] The third selector is used to select the input of the second output value of the third selector to the second selector;

[0281] The second selector is used to select which output value to send to the adder;

[0282] The adder is used to accumulate the second output value and the seventh input data to obtain the accumulated value of the second output value and the seventh input data;

[0283] The second output interface is used to output the sum of the second output value and the seventh input data.

[0284] In one possible implementation, the structure of the convolutional neural network model includes convolutional layers; determining a plurality of target reconfigurable computational units from the plurality of reconfigurable computational units based on the structure of the convolutional neural network model to be processed includes:

[0285] Based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from the plurality of reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix. 3 The first objective is a reconfigurable computing unit, according to n 2 Arranged in rows and n columns, where n is a positive integer;

[0286] The step of configuring the computation types of the plurality of target reconfigurable computation units and the connection relationships between the plurality of target reconfigurable computation units according to the structure of the convolutional neural network model includes:

[0287] Configure n according to the convolution kernel matrix. 3 The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of the n second target reconfigurable computing units is the accumulation process.

[0288] Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

[0289] The convolutional neural network computing device provided in this application embodiment is used to execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0290] Figure 12 An example is a schematic diagram of the physical structure of a mobile terminal, such as... Figure 12As shown, the mobile terminal may include: a processor 1210, a communications interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 communicate with each other through the communication bus 1240. Processor 1210 can call logic instructions in memory 1230 to execute a convolutional neural network (CNN) operation method. This method is applied to a CNN operation device configured with multiple reconfigurable computing units. The method includes: determining multiple target reconfigurable computing units from the multiple reconfigurable computing units based on the structure of the CNN model to be processed; configuring the computation type of each of the multiple target reconfigurable computing units and the connection relationship between the multiple target reconfigurable computing units based on the structure of the CNN model, wherein the computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing; broadcasting data in the CNN model to the corresponding target reconfigurable computing units for computation based on the computation type of each target reconfigurable computing unit, obtaining the computation results of each of the multiple target reconfigurable computing units, wherein at least one piece of data in the CNN model is broadcast to at least two target reconfigurable computing units; and processing the computation results of each of the multiple target reconfigurable computing units based on the connection relationship between the multiple target reconfigurable computing units to obtain the model output result of the CNN model.

[0291] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0292] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the convolutional neural network operation method provided in the above embodiments. This method is applied to a convolutional neural network operation device, which is configured with multiple reconfigurable computing units. The method includes: determining multiple target reconfigurable computing units from the multiple reconfigurable computing units according to the structure of the convolutional neural network model to be processed; configuring the computation type of each of the multiple target reconfigurable computing units and the multiple target reconfigurable computing units according to the structure of the convolutional neural network model. The connection relationships between reconfigurable computing units are defined, wherein the computing type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing. Based on the computing type of each of the multiple target reconfigurable computing units, the data in the convolutional neural network model is broadcast to the corresponding target reconfigurable computing unit for computation, resulting in the computation results of each of the multiple target reconfigurable computing units. Specifically, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units. Based on the connection relationships between the multiple target reconfigurable computing units, the computation results of each of the multiple target reconfigurable computing units are processed to obtain the model output of the convolutional neural network model.

[0293] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program is implemented to perform the convolutional neural network operation method provided in the above embodiments. This method is applied to a convolutional neural network operation device configured with multiple reconfigurable computing units. The method includes: determining multiple target reconfigurable computing units from the multiple reconfigurable computing units according to the structure of the convolutional neural network model to be processed; configuring the computation type of each of the multiple target reconfigurable computing units and the connection relationship between the multiple target reconfigurable computing units according to the structure of the convolutional neural network model. The calculation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing. Based on the calculation type of each of the multiple target reconfigurable computing units, the data in the convolutional neural network model is broadcast to the corresponding target reconfigurable computing unit for calculation, obtaining the calculation results of each of the multiple target reconfigurable computing units. Specifically, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units. Based on the connection relationship between the multiple target reconfigurable computing units, the calculation results of each of the multiple target reconfigurable computing units are processed to obtain the model output result of the convolutional neural network model.

[0294] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0295] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0296] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A convolutional neural network computing device, characterized in that, The convolutional neural network computing device is configured with multiple reconfigurable computing units, and the device includes: The determining module is used to determine multiple target reconfigurable computing units from the multiple reconfigurable computing units based on the structure of the convolutional neural network model to be processed; The configuration module is used to configure the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units according to the structure of the convolutional neural network model. The computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing. The broadcast module is used to broadcast the data in the convolutional neural network model to the corresponding target reconfigurable computing units for calculation according to the calculation type of each of the multiple target reconfigurable computing units, so as to obtain the calculation results of each of the multiple target reconfigurable computing units. In this case, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units. The calculation module is used to process the calculation results of each of the multiple target reconfigurable calculation units according to the connection relationship between the multiple target reconfigurable calculation units, so as to obtain the model output result of the convolutional neural network model; The reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein: The input interface includes a first input interface, a second input interface, a third input interface, a fourth input interface, a fifth input interface, and a sixth input interface; The output interface includes a first output interface and a second output interface; The computational subunit includes a comparator, a multiplier, an adder, a first gating unit, a second gating unit, and a third gating unit; The input terminal of the comparator is connected to the first input interface and the second input interface, and the output terminal of the comparator is connected to the first output interface; The input terminal of the multiplier is connected to the third input interface and the fourth input interface, and the output terminal of the multiplier is connected to the input terminal of the first selector and the first output interface; The input terminal of the first selector is also connected to the fifth input interface, the input terminal of the second selector is connected to the sixth input interface, and the output terminals of the first selector and the second selector are both connected to the input terminal of the adder. The output of the adder is connected to the input of the third selector; The output terminal of the third selector is connected to the input terminal of the second selector and the second output interface; The structure of the convolutional neural network model includes convolutional layers; the determining module is specifically used for: Based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from the plurality of reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix. 3 The first objective is a reconfigurable computing unit, in order of n 2 Arranged in rows and n columns, where n is a positive integer; The configuration module is specifically used for: Configure n according to the convolution kernel matrix. 3 The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of the n second target reconfigurable computing units is the accumulation process. Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

2. The apparatus according to claim 1, characterized in that, The calculation type is a comparison process; wherein: The first input interface is used to acquire first input data and send the first input data to the comparator; The second input interface is used to acquire second input data and send the second input data to the comparator; The comparator is used to compare the first input data and the second input data to determine the larger value between the first input data and the second input data. The first output interface is used to output the larger value between the first input data and the second input data.

3. The apparatus according to claim 1, characterized in that, The calculation type is either multiplication or multiply-accumulate; wherein: The third input interface is used to acquire third input data and send the third input data to the multiplier; The fourth input interface is used to acquire fourth input data and send the fourth input data to the multiplier; The multiplier is used to perform multiplication on the third input data and the fourth input data to obtain the product of the third input data and the fourth input data.

4. The apparatus according to claim 3, characterized in that, If the calculation type is the multiplication operation, then: The multiplier is also used to send the product to the first output interface; The first output interface is used to output the product; If the calculation type is the multiply-accumulate process, then: The multiplier is also used to send the product to the first gate; The first gate is used to select which product to send to the adder; The third selector is used to select the first output data selected by the third selector to be input into the second selector; The second selector is used to select which output data to send to the adder; The adder is used to add the first output data and the product to obtain the sum of the first output data and the product; The second output interface is used to output the sum of the first output data and the product.

5. The apparatus according to claim 1, characterized in that, The calculation type is addition; wherein: The fifth input interface is used to acquire fifth input data and send the fifth input data to the first selector; The first gate is used to select which input data to send to the adder; The sixth input interface is used to acquire a sixth input value and send the sixth input value to the second selector; The second gate is used to select which input data to send to the adder; The adder is used to perform addition on the fifth input data and the sixth input data to obtain the sum of the fifth input data and the sixth input data; The second output interface is used to output the sum of the fifth input data and the sixth input data.

6. The apparatus according to claim 1, characterized in that, The calculation type is accumulation processing; wherein: The fifth input interface is used to acquire the seventh input data and send the seventh input data to the first selector; The first selector is used to select which input data to send to the adder; The third selector is used to select the input of the second output value of the third selector to the second selector; The second selector is used to select which output value to send to the adder; The adder is used to accumulate the second output value and the seventh input data to obtain the accumulated value of the second output value and the seventh input data; The second output interface is used to output the sum of the second output value and the seventh input data.

7. A convolutional neural network operation method, characterized in that, Applied to a convolutional neural network computing device, wherein the convolutional neural network computing device is configured with multiple reconfigurable computing units, the method includes: Based on the structure of the convolutional neural network model to be processed, multiple target reconfigurable computing units are determined from the multiple reconfigurable computing units; Based on the structure of the convolutional neural network model, the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units are configured. The computation type is any one of comparison processing, addition processing, multiplication processing, accumulation processing, and multiply-accumulate processing. According to the calculation type of each of the plurality of target reconfigurable computing units, the data in the convolutional neural network model is broadcast to the corresponding target reconfigurable computing unit for calculation, and the calculation results of each of the plurality of target reconfigurable computing units are obtained. In this case, at least one piece of data in the convolutional neural network model is broadcast to at least two target reconfigurable computing units. Based on the connection relationship between the multiple target reconfigurable computing units, the calculation results of each of the multiple target reconfigurable computing units are processed to obtain the model output result of the convolutional neural network model; The reconfigurable computing unit includes an input interface, an output interface, and a computing subunit, wherein: The input interface includes a first input interface, a second input interface, a third input interface, a fourth input interface, a fifth input interface, and a sixth input interface; The output interface includes a first output interface and a second output interface; The computational subunit includes a comparator, a multiplier, an adder, a first gating unit, a second gating unit, and a third gating unit; The input terminal of the comparator is connected to the first input interface and the second input interface, and the output terminal of the comparator is connected to the first output interface; The input terminal of the multiplier is connected to the third input interface and the fourth input interface, and the output terminal of the multiplier is connected to the input terminal of the first selector and the first output interface; The input terminal of the first selector is also connected to the fifth input interface, the input terminal of the second selector is connected to the sixth input interface, and the output terminals of the first selector and the second selector are both connected to the input terminal of the adder. The output of the adder is connected to the input of the third selector; The output terminal of the third selector is connected to the input terminal of the second selector and the second output interface; The structure of the convolutional neural network model includes convolutional layers; the step of determining multiple target reconfigurable computational units from the multiple reconfigurable computational units based on the structure of the convolutional neural network model to be processed includes: Based on the dimension of the convolutional kernel matrix of the convolutional layer, n is determined from the plurality of reconfigurable computational units. 3 There are n first-objective reconfigurable computational units and n second-objective reconfigurable computational units, where n is the dimension of the convolution kernel matrix. 3 The first objective is a reconfigurable computing unit, in order of n 2 Arranged in rows and n columns, where n is a positive integer; The step of configuring the computation type of each of the plurality of target reconfigurable computation units and the connection relationship between the plurality of target reconfigurable computation units according to the structure of the convolutional neural network model includes: Configure n according to the convolution kernel matrix. 3 The computation type of each of the first target reconfigurable computing units is multiplication, and the computation type of the n second target reconfigurable computing units is the accumulation process. Based on the convolution kernel matrix, any first target reconfigurable computational unit in the i-th column is configured to be connected to the i-th second target reconfigurable computational unit.

8. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the convolutional neural network operation method as described in claim 7.

Images