A data processing method, device, chip, equipment and medium

Abstract

The present application discloses a data processing method, device, chip, equipment and medium to solve the problem of slow computing speed of neural network in the prior art. In the present application, the electronic device can determine the second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer; determine the third size of the input data according to the second size of the output data; divide the data in the external memory according to the third size of the input data and the number of the convolution kernels, and input the divided data into the memory, so that there is no need to carry out data transfer every time calculation is performed, which can reduce the number of data transfers and the transfer repetition rate, and can improve the data reuse rate, thereby improving the computing speed of the neural network.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a data processing method, device, chip, equipment and medium. Background Art

[0002] As chip computing power increases, each neural network layer, especially in image processing neural networks, will have a large amount of feature data and convolution kernel data input and output. A large amount of feature data and convolution kernel data are usually stored in external memory with large storage space but slow reading speed. Each time they are used, they need to be moved from the external memory to the internal memory with small storage space but fast reading speed for the chip to use.

[0003] Due to the limited memory size, it may not be possible to move the feature data and convolution kernel data in the external memory to the memory at once. Generally, in this case, according to the order in which the chip uses the data, the feature data and convolution kernel data currently required for input of this layer are first obtained from the external memory, which may include the feature data output by the previous neural network layer, and then moved to the memory for calculations such as multiplication and addition calculation units, activation calculation units, and pooling calculation units. Then, the output feature data is moved to the external memory as the input of the next neural network layer, and repeated many times until the calculation of all neural network layers is completed.

[0004] It can be seen that the required feature data and convolution kernel data are obtained from the external memory for each calculation, and the output feature data is saved in the external memory after each calculation. The amount of feature data and convolution kernel data to be transported is large, resulting in a slower calculation speed of the neural network. Summary of the invention

[0005] The present application provides a data processing method, device, chip, equipment and medium to solve the problem of slow computing speed of neural networks in the prior art.

[0006] The present application provides a data processing method, the method comprising:

[0007] Determine a second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernels in the neural network layer;

[0008] Determining a third size of the input data according to the second size of the output data;

[0009] The data in the external memory is segmented according to the third size of the input data and the number of the convolution kernels, and the segmented data is input into the internal memory.

[0010] Further, determining the second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0011] Determining a first data amount corresponding to the group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels;

[0012] Determine the initial size of the input data according to the initial size of the output data;

[0013] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0014] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0015] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

[0016] Furthermore, determining the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0017] According to the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the groups is determined.

[0018] Furthermore, the data in the external memory is segmented, including:

[0019] The data in the external memory is rearranged, and the rearranged data is segmented.

[0020] Further, the data includes characteristic data, and the rearranging of the data in the external memory includes:

[0021] Sorting the feature data according to the input order, and grouping the sorted feature data according to a first set number;

[0022] In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

[0023] Further, the data includes convolution kernel data, and the rearranging of the data in the external memory includes:

[0024] Sorting the convolution kernel data according to an input order, and grouping the sorted convolution kernel data according to a second set number;

[0025] In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are sorted according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0026] The present application provides a data processing device, the device comprising:

[0027] A first determination module, used to determine a second size of output data and the number of convolution kernels according to a storage space of the memory and a first size of the convolution kernel in the neural network layer;

[0028] A second determining module, configured to determine a third size of the input data according to the second size of the output data;

[0029] The segmentation module is used to segment the data in the external memory according to the third size of the input data and the number of the convolution kernels, and input the segmented data into the internal memory.

[0030] An embodiment of the present application also provides a chip, which is used to execute the steps of any of the above-mentioned data processing methods.

[0031] An embodiment of the present application also provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned data processing methods when executing a computer program stored in the memory.

[0032] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0033] In the embodiment of the present application, the size of the input data and the number of convolution kernels are obtained according to the storage space of the memory and the first size of the convolution kernel in the neural network layer, and then the data in the external memory is divided and moved. There is no need to move the data every time calculation is performed, which can reduce the number of data moves and the repetition rate of the moves, and can increase the data reuse rate, thereby improving the calculation speed of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of a data processing process provided for some embodiments of the present application;

[0036] Figure 2 A schematic diagram of a data processing flow provided for some embodiments of the present application;

[0037] Figure 3 A schematic diagram of a data processing process provided for some embodiments of the present application;

[0038] Figure 4 A schematic diagram of a data rearrangement process provided for some embodiments of the present application;

[0039] Figure 5 A schematic diagram of a data rearrangement process provided for some embodiments of the present application;

[0040] Figure 6 A schematic diagram of the structure of a data processing device provided in some embodiments of the present application;

[0041] Figure 7 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0043] In order to improve the computing speed of a neural network, the embodiments of the present application provide a data processing method, apparatus, chip, device and medium.

[0044] Embodiment 1:

[0045] Figure 1 A schematic diagram of a data processing process provided in an embodiment of the present application, the process includes the following steps:

[0046] S101: Determine a second size of output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer.

[0047] The data processing method provided in the embodiment of the present application is applied to an electronic device or a chip or a chip in an electronic device, specifically, it can be applied to a processing unit such as a neural-network processing unit (NPU) in an electronic device. It is understandable that in some cases, the NPU can also be a chip. The electronic device includes but is not limited to an image acquisition device and an attendance machine. For example, the image acquisition device can be a camera, and the electronic device can be applied to a door access control scene or an attendance scene.

[0048] The electronic device may be provided with external memory and internal memory, which are used to store the data required in the neural network calculation process. The storage space of the external memory is large but the reading speed is slow, and the storage space of the internal memory is small but the reading speed is fast, so the internal memory usually stores the data required for the neural network to input this calculation and the data output of this calculation. For ease of description, in the embodiment of the present application, the data required for input of a calculation is referred to as input data, and the data output of a calculation is referred to as output data. In the embodiment of the present application, the memory type of the external memory and the internal memory, as well as the size of the storage space are not limited.

[0049] Among them, the data includes feature data and / or convolution kernel data. Feature data refers to feature maps, and convolution kernel data includes weight data (or called convolution weights). The "neural network" involved in the embodiments of the present application refers to a neural network including a convolution layer, such as a convolutional neural network (CNN). The calculation process of the neural network is completed by an accelerator in an electronic device. The accelerator generally refers to a hardware accelerator, which is deployed in the form of a processing unit group.

[0050] In the best case, the feature data and convolution kernel data are moved from the external memory to the internal memory at one time, that is, the feature data and the convolution kernel data are moved only once, which can achieve maximum reuse of the feature data and the convolution kernel data. However, in some cases, the storage space of the internal memory is limited, and it is impossible to read all the feature data and the convolution kernel data at one time. Therefore, the feature data and / or the convolution kernel data can be segmented, and then the partial data blocks obtained by segmentation are moved to the internal memory. The embodiments of this application are mainly explained for this situation.

[0051] In a calculation process, the memory storage space generally needs to store the input data required for the calculation, the output data of the calculation, and the convolution kernel. In other words, the sum of the storage space occupied by the input data, output data, and convolution kernel cannot exceed the memory storage space. In addition, the size of the memory storage space and the first size of the convolution kernel in the neural network layer are fixed. Therefore, according to these two fixed parameters, the second size of the output data, the number of convolution kernels, and the third size of the input data are determined to divide the data in the external memory. Under the premise of meeting the memory requirements, as many convolution kernels and feature data as possible can be read into the memory to reduce the number of data transfers and the transfer repetition rate, and to increase the data reuse rate, thereby improving the calculation speed of the neural network.

[0052] In S101, the initial number of convolution kernels and the initial size of output data can be preset, and the size of input data can be determined based on the initial size of output data, and the data volume of the convolution kernel (i.e., the storage space occupied by the convolution kernel) can be determined based on the initial number of convolution kernels and the first size of the convolution kernel. If the sum of the initial size of the output data, the determined size of the input data and the determined data volume of the convolution kernel does not exceed the storage space of the memory, the initial number of convolution kernels can be determined as the number of convolution kernels, and the initial size of the output data can be determined as the second size of the output data; alternatively, the initial size of the output data and the number of convolution kernels can be adjusted so that as many convolution kernels and input data as possible can be read in without exceeding the storage space of the memory.

[0053] The first size of the convolution kernel in each neural network layer in the neural network may be the same or different. If the first size of the convolution kernel in each neural network layer in the neural network is the same, the electronic device executes S101 once to determine the second size of the output data corresponding to each neural network layer and the number of convolution kernels. If the first size of the convolution kernel in each neural network in the neural network is different, then in S101, the electronic device may determine the second size of the output data and the number of convolution kernels for each neural network layer according to the memory space of the memory and the first size of the convolution kernel in the neural network layer.

[0054] S102: Determine a third size of the input data according to the second size of the output data.

[0055] The second size of the output data is related to the height of the output data and the width of the output data, and the third size of the input data is related to the height of the input data and the width of the input data.

[0056] The height of the input data is related to the height of the output data. Optionally, the height of the input data can also be related to the step size of the convolution kernel and the first size of the convolution kernel (such as the height of the convolution kernel). In one possible implementation, in_h = (out_h-1)*filter_sh+filter_h, where in_h is the height of the input data, out_h is the height of the output data, filter_sh is the step size of the convolution kernel moving in the height direction, and filter_h is the height of the convolution kernel.

[0057] The width of the input data is related to the width of the input data. Optionally, the width of the input data can also be related to the step size of the convolution kernel and the first size of the convolution kernel (such as the width of the convolution kernel). In one possible implementation, in_w = (out_w-1)*filter_sw+filter_w, where in_w is the width of the input data, out_w is the width of the output data, filter_sw is the step size of the convolution kernel moving in the width direction, and filter_w is the width of the convolution kernel.

[0058] Optionally, the third size of the input data may also be related to the number of channels of the convolution kernel. In a possible implementation, the third size of the input data in_size=in_h*in_w*C, where C is the number of channels of the convolution kernel.

[0059] S103: Split the data in the external memory according to the third size of the input data and the number of convolution kernels, and input the split data into the internal memory.

[0060] In S103, the electronic device can determine the amount of data input into the memory based on the third size of the input data and the number of convolution kernels, and then divide the data in the external memory to obtain data of the amount, and input the divided data into the memory.

[0061] In one example, the convolution kernel data can be moved to the memory at one time, and the feature data in the external memory can be segmented. At this time, the multiple first data blocks obtained include multiple first feature data blocks.

[0062] like Figure 2 As shown in the figure, the convolution kernel data and the input feature data are stored in the external memory. The convolution kernel data can be moved to the internal memory at one time, but the amount of input feature data is too large to be moved to the internal memory at one time. Therefore, part of the input feature data can be moved to the internal memory by segmentation. The neural network performs convolution and other calculations on the convolution kernel data and part of the input feature data in the internal memory, and then obtains the corresponding part of the output feature data, which is then stored in the external memory for further storage. Figure 2 It can be seen that the convolution kernel data and input feature data involved in the calculation, as well as the corresponding output feature data, will occupy storage space in the memory.

[0063] In another example, the feature data and convolution kernel data in the external memory may be divided separately, and the multiple first data blocks obtained at this time include multiple first feature data blocks and multiple first convolution kernel data blocks.

[0064] In the embodiment of the present application, the size of the input data and the number of convolution kernels are obtained according to the storage space of the memory and the first size of the convolution kernel in the neural network layer, and then the data in the external memory is divided and moved. There is no need to move the data every time calculation is performed, which can reduce the number of data moves and the repetition rate of the moves, and can increase the data reuse rate, thereby improving the calculation speed of the neural network.

[0065] Embodiment 2:

[0066] In order to further improve the computing speed of the neural network, based on the above embodiment, in the embodiment of the present application, the second size of the output data and the number of convolution kernels are determined according to the storage space of the memory and the first size of the convolution kernel in the neural network layer, including:

[0067] Determining a first data amount corresponding to a group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels;

[0068] Determine the initial size of the input data according to the initial size of the output data;

[0069] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0070] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0071] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

[0072] Under the premise of not exceeding the storage space of the memory, reading as much feature data as possible into the memory can reduce the number of data transfers and the duplication rate, increase the data reuse rate, and further improve the calculation speed of the neural network.

[0073] The number of a group of convolution kernels is also called the number of unit convolution kernels. The number of a group of convolution kernels may be related to the number of processing unit groups of the hardware accelerator. For example, if the number of processing unit groups Atomic-C is 8, the number of a group of convolution kernels is 8, that is, the number of unit convolution kernels is 8, that is, each group of convolution kernels includes 8 convolution kernels. The number of a group of convolution kernels can be regarded as the initial number of preset convolution kernels, and can also be used as the second channel number of output data.

[0074] The first size of the convolution kernel is related to the height and width of the convolution kernel.

[0075] The first data volume corresponding to a group of convolution kernels is the storage space occupied by a group of convolution kernels. In one possible implementation, A_filter_size = filter_h*filter_w*C*A, where A_filter_size is the first data volume corresponding to a group of convolution kernels, filter_h is the height of the convolution kernel, filter_w is the width of the convolution kernel, C is the first channel number of the convolution kernel, and A is the number of a group of convolution kernels or the second channel number of the output data. Optionally, the number of a group of convolution kernels here can be the default minimum number, and the number of convolution kernels can be expanded later.

[0076] The value of the initial size of the output data is not limited here. Optionally, the initial size of the output data may be related to the hardware accelerator. For example, the initial size of the output data may be set to minimize the amount of data transported while maximizing the use of the hardware accelerator. The process of the electronic device determining the initial size of the input data based on the initial size of the output data is similar to the above S102 and is not described here.

[0077] The second data amount corresponding to a calculation is the storage space occupied by a calculation. In a possible implementation, total_size=in_size+A_filter_size+out_size, where total_size is the second data amount corresponding to a calculation, in_size is the initial size of the output data, out_size is the initial size of the input data, and A_filter_size is the first data amount.

[0078] The electronic device also needs to judge whether the second data amount meets the requirements, and the judgment needs to meet the following two conditions at the same time: 1) the second data amount is smaller than the storage space of the memory, and 2) the initial size of the input data is smaller than the set size. The value of the set size is not limited in the embodiment of the present application. If any of the above two conditions is not met, the electronic device can adjust the initial size of the output data and recalculate the second data amount until the adjusted second data amount and the adjusted initial size of the output data meet the above two conditions. When the above two conditions are met at the same time, the initial size of the output data is determined as the second size of the output data.

[0079] In the embodiment of the present application, without exceeding the storage space of the memory, as much feature data as possible is read into the memory, which can reduce the number of data transfers and the transfer repetition rate, increase the data reuse rate, and thus further improve the calculation speed of the neural network.

[0080] Embodiment 3:

[0081] In order to further improve the computing speed of the neural network, based on the above embodiments, in the embodiment of the present application, the number of convolution kernels is determined according to the storage space of the memory and the first size of the convolution kernel in the neural network layer, including:

[0082] According to the storage space of the memory, the second size of the output data and the first data amount corresponding to a group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the group is determined.

[0083] The more convolution kernels read in at one time, the higher the reuse rate of the convolution kernel data, the fewer times the feature data is repeatedly transferred, and the faster the calculation speed of the neural network. Therefore, in the embodiment of the present application, the number of convolution kernels can also be expanded as a unit, that is, the number of expanded convolution kernels is an integer multiple of the unit convolution kernel.

[0084] Considering that the number of convolution kernels and the size of output data are interrelated and proportional, the size of the input data can be divided by the storage space of the memory. The remaining storage space can include the number of unit convolution kernels and output data, and the number of groups of convolution kernels that can be expanded can be obtained, and the size of the divided convolution kernel can be obtained.

[0085] In a possible implementation, group_filter = (memory_size-in_size) / (A_filter_size+out_size), where group_filter is the number of groups of convolution kernels, memory_size is the storage space of the memory. group_filter*A is the number of convolution kernels in the group, and group_filter*A_filter_size is the amount of data of the convolution kernel in the group.

[0086] In another possible implementation, the electronic device can obtain the total number of convolution kernels that are not actually split, and thus can obtain the number of groups of convolution kernels that can actually be expanded. In this way, the electronic device determines the number of convolution kernels that can be read into the memory for this calculation based on the number of groups of convolution kernels that can actually be expanded and the number of groups of convolution kernels that can be expanded with the remaining storage space.

[0087] For example, real_group_filter = num_filter / A, where real_group_filter is the number of groups of convolution kernels that can actually be expanded, and num_filter is the total amount of data of the undivided convolution kernels. When real_group_filter is smaller than group_filter, the remaining storage space in the memory can read in the undivided convolution kernels at one time. At this time, the total amount of data of the undivided convolution kernels, num_filter, can be used as the amount of data of the convolution kernels read into the memory, filter_size, and real_group_filter*A is the number of convolution kernels of this group. When real_group_filter is greater than or equal to group_filter, the remaining storage space in the memory cannot read in all the undivided convolution kernels at one time. At this time, group_filter*A can be used as the number of convolution kernels of this group, and group_filter*A_filter_size is the amount of data of the convolution kernels of this group.

[0088] The following is a specific example, see Figure 3 As shown, the following steps are included:

[0089] S301: Determine a first data amount corresponding to a unit convolution kernel.

[0090] In S301, data segmentation is first performed according to the unit convolution kernel, and convolution and quantity expansion are performed in subsequent steps.

[0091] For example, the first data amount corresponding to the unit convolution kernel is A_filter_size=filter_h*filter_w*C*A.

[0092] S302: Determine the height and width of the input feature data.

[0093] For example, the width of the input feature data in_w = (out_w-1)*filter_sw+filter_w, and the height of the input feature data in_h = (out_h-1)*filter_sh+filter_h.

[0094] S303: Determine the size of the input feature data and the size of the output feature data.

[0095] For example, the size of the input feature data in_size=in_h*in_w*C, and the size of the output feature data out_size=out_h*out_w*A.

[0096] S304: Determine a second data volume corresponding to one calculation.

[0097] For example, the second data amount corresponding to one calculation is total_size=in_size+A_filter_size+out_size.

[0098] S305: Determine whether the following conditions are met at the same time: the second data volume is smaller than the storage space of the memory and the size of the input feature data is smaller than the set size; if yes, execute S306; if not, update the height and width of the output feature data and return to S302.

[0099] That is to say, when any of the following conditions are not met, the height and width of the output feature data are adjusted and updated until the above conditions are met at the same time. However, when the above conditions are met at the same time, the number of convolution kernels can be adjusted on this basis to maximize the number of convolution kernels read in each time, thereby maximizing the reuse rate of convolution kernel data.

[0100] S306: Expand the number of convolution kernels to determine the number of convolution kernels read into the memory.

[0101] For example, the number of convolution kernels is expanded by an integer multiple of the unit convolution kernel. Assuming that the number of groups of convolution kernels that can be read in by the remaining storage space in the memory is group_filter = (memory_size-in_size) / (A_filter_size+out_size), the number of groups of undivided convolution kernels that can actually be expanded is real_group_filter = num_filter / A. When real_group_filter is less than group_filter, the number of groups of convolution kernels read into the memory is real_group_filter, and the number of convolution kernels read into the memory is num_filter. When real_group_filter is greater than or equal to group_filter, the number of groups of convolution kernels read into the memory is group_filter, and the amount of convolution kernel data read into the memory is group_filter*A_filter_size.

[0102] S307: Determine the size of the adjusted input data and the amount of convolution kernel data read into the memory.

[0103] In S307 , the size of the adjusted output data may also be determined.

[0104] In the embodiment of the present application, the number of convolution kernels is expanded in groups. The more convolution kernels are read in at one time, the higher the reuse rate of the convolution kernel data is, and the fewer times the feature data is repeatedly transferred, which can further improve the calculation speed of the neural network.

[0105] Embodiment 4:

[0106] In order to further ensure the correctness of data segmentation, based on the above embodiments, in the embodiment of the present application, the data in the external memory is segmented, including:

[0107] The data in the external memory is rearranged, and the rearranged data is segmented.

[0108] The data in the external memory is rearranged (reshaped). The rearranged data can further ensure the correctness of data segmentation and improve the resource utilization of the hardware accelerator.

[0109] In one example, the data includes feature data, and rearranging the data in the external memory includes: sorting the feature data in an input order, and grouping the sorted feature data according to a first set number; in each group of feature data, rearranging the pixels in the feature data in the group of feature data according to the input order of the feature data and the arrangement order of the pixels.

[0110] The first set number may be related to the number of processing unit groups, and the value of the first set number is not limited. For example, in each group of feature data, the first pixel point of each feature data and the second pixel point of each feature data may be taken respectively according to the input order of the feature data, until all the pixels of each feature data are taken. If the number of feature data in the group does not reach the first set number, the first value may be used to fill it, and the first value may be any value, for example, the first value may be 0.

[0111] For example Figure 4 As shown, the feature data are grouped according to the number of processing unit groups (for example, A), that is, each group of feature data includes A feature data. For the feature data of the first group, from the first pixel point (for example, Figure 4 Starting from the pixel 0 in , take the first pixel of the first feature data (i.e., the first feature map), the first pixel of the second feature data, ..., the first pixel of the Ath feature data, and so on, take the second pixel of the 1st to Ath feature data (i.e., Figure 4 1), until all the pixel points of the feature data in the first group are taken, that is, the 16th pixel points of the feature data from 1 to A are taken (such as Figure 4 15), and so on, until all the pixel points of all groups of feature data are obtained.

[0112] exist Figure 4 In the memory, according to the arrangement order of the pixels, the addresses occupied in sequence are from low (Low address) to high (High address). The 16 pixels of the A feature data in the first group are pixel 0 to pixel 15, and the 16 pixels of the A feature data in the second group are pixel 16 to pixel 31. The address corresponding to the first pixel of the A feature data in the first group is I0_0, I0_1, ..., I0_(A-1), the address corresponding to the second pixel to the 16th pixel of the A feature data is I1_0, I1_1, ..., I15_(A-1), the address corresponding to the first pixel of the A feature data in the second group is I16_0, I16_1, ..., I16_(A-1), and the address corresponding to the second pixel to the 16th pixel of the A feature data is I17_0, I17_1, ..., I31_(A-1). If there are more than A pieces of feature data, the feature data can be grouped according to multiples of A. If there are less than A pieces of feature data in a group, they can be padded with 0.

[0113] In another example, the data includes convolution kernel data, and rearranging the data in the external memory includes: sorting the data according to an input order of the convolution kernel data, and grouping the sorted convolution kernel data according to a second set number; in each group of convolution kernel data, sorting the pixels in the convolution kernel data in the group of convolution kernel data according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0114] The second set number may be related to the number of processing unit groups, and the second set number may be the same or different from the first set number, and there is no restriction on the value of the second set number. For example, in each group of feature data, all pixel points of each channel may be taken for each channel according to the input order of the channels of the convolution kernel, until all pixel points of all channels of all convolution kernel data are taken. Optionally, the channels of each pixel point may also be grouped according to a third set number, and the third set number is related to the number of channels of the convolution kernel. There is no restriction on the value of the third set number, for example, the third set number is 8. If the number of convolution kernels in the group does not reach the second set number, the second numerical value may be used to fill the gap, and the second numerical value may be any value, and the second numerical value may be the same or different from the first numerical value, for example, the second numerical value may be 0.

[0115] For example Figure 5 As shown, the convolution kernel data are grouped according to the number of processing unit groups (for example, A), that is, each group of feature data includes A convolution kernel data (corresponding to A convolution kernels), Figure 5 A set of feature data includes convolution kernel data K0, K1, ..., K(A-1). Starting from the first pixel, take the first channel of the first pixel of the first channel of the first convolution kernel data, ..., the eighth channel of the first pixel of the first channel of the first convolution kernel data (the third set number is 8 at this time, that is, the depth or width of the pixel in a convolution kernel is 8), and then take the first channel of the first pixel of the first channel of the second convolution kernel data, and so on, take the 8 channels of the first pixel of the first channel of A convolution kernel data, and in this way, take the 8 channels of all pixel points of the first channel of A convolution kernel data, and then take the 8 channels of all pixel points of all channels of A convolution kernel data. Assume that each channel of the convolution kernel data has B pixels, and each pixel corresponds to 8 channels.

[0116] exist Figure 5In the order of arrangement of pixels and channels, the addresses occupied in memory are from low (Lowaddress) to high (Highaddress). address), the addresses corresponding to the 8 channels of the 1st pixel of the 1st channel of the 1st convolution kernel data are K0_0_0_0, …, K0_0_0_7, the addresses corresponding to the 8 channels of the 1st pixel of the 1st channel of the 2nd convolution kernel data to the 8 channels of the 1st pixel of the 1st channel of the Ath convolution kernel data are K1_0_0_0, …, K(A-1)_0_0_7, the addresses corresponding to the 8 channels of the 2nd pixel of the 1st channel of the Ath convolution kernel data to the 8 channels of the Bth pixel of the 1st channel of the Ath convolution kernel are K0_0_1_0, …, K(A-1)_0_(B-1)_7, and the addresses corresponding to the 8 channels of the B pixel of the 2nd channel of the Ath convolution kernel to the B pixel of the 9th channel of the Ath convolution kernel are K0_1_1_0, …, K(A-1)_8_(B-1)_7. If there are more than A convolution kernel data, the convolution kernel data can be grouped according to multiples of A. If there are less than A convolution kernels in a group, they can be padded with 0.

[0117] In the embodiment of the present application, by rearranging the data in the external memory, the correctness of the rearranged data during segmentation can be further guaranteed, and the resource utilization of the hardware accelerator can be further improved.

[0118] In order to improve the calculation speed of the neural network, on the basis of the above embodiments and based on the same technical concept as the above method embodiments, the present application also provides a data processing device. Figure 6 A structural diagram of a data processing device 600 provided in this application is shown in FIG. Figure 6 As shown, the data processing device 600 includes:

[0119] A first determination module 601 is used to determine a second size of output data and the number of convolution kernels according to a storage space of the memory and a first size of the convolution kernel in the neural network layer;

[0120] A second determining module 602, configured to determine a third size of the input data according to the second size of the output data;

[0121] The segmentation module 603 is used to segment the data in the external memory according to the third size of the input data and the number of the convolution kernels, and input the segmented data into the internal memory.

[0122] In a possible implementation manner, the first determining module 601 is specifically configured to:

[0123] Determining a first data amount corresponding to the group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels;

[0124] Determine the initial size of the input data according to the initial size of the output data;

[0125] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0126] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0127] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, determining the initial size of the output data as the second size of the output data;

[0128] In a possible implementation manner, the first determining module 601 is specifically configured to:

[0129] According to the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the groups is determined.

[0130] In a possible implementation manner, the segmentation module 603 is specifically configured to:

[0131] The data in the external memory is rearranged, and the rearranged data is segmented.

[0132] In a possible implementation, the data includes feature data, and the segmentation module 603 is specifically configured to:

[0133] Sorting the feature data according to the input order, and grouping the sorted feature data according to a first set number;

[0134] In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

[0135] In a possible implementation, the data includes convolution kernel data, and the segmentation module 603 is specifically used to:

[0136] Sorting the convolution kernel data according to an input order, and grouping the sorted convolution kernel data according to a second set number;

[0137] In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are rearranged according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0138] Figure 7 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the present application embodiment further provides an electronic device, such as Figure 7 As shown, it includes: a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704;

[0139] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 performs the following steps:

[0140] Determine a second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernels in the neural network layer;

[0141] Determining a third size of the input data according to the second size of the output data;

[0142] The data in the external memory is segmented according to the third size of the input data and the number of the convolution kernels, and the segmented data is input into the internal memory.

[0143] Further, the processor 701 is specifically configured to determine a first amount of data corresponding to the group of convolution kernels according to the first size of the convolution kernel, the first number of channels of the convolution kernel, and the number of the group of convolution kernels;

[0144] Determine the initial size of the input data according to the initial size of the output data;

[0145] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0146] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0147] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

[0148] Furthermore, the processor 701 is specifically used to determine the number of groups of convolution kernels and the number of convolution kernels in the groups based on the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels.

[0149] Furthermore, the processor 701 is specifically configured to rearrange the data in the external memory and segment the rearranged data.

[0150] Further, the data includes feature data, and the processor 701 is specifically configured to sort the feature data according to an input order of the feature data, and group the sorted feature data according to a first set number;

[0151] In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

[0152] The data includes convolution kernel data. Further, the processor 701 is specifically configured to sort the convolution kernel data according to an input order of the convolution kernel data, and group the sorted convolution kernel data according to a second set number;

[0153] In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are rearranged according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0154] The communication interface 702 is used for communication between the electronic device and other devices.

[0155] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0156] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0157] On the basis of the above embodiments, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0158] Determine a second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernels in the neural network layer;

[0159] Determining a third size of the input data according to the second size of the output data;

[0160] The data in the external memory is segmented according to the third size of the input data and the number of the convolution kernels, and the segmented data is input into the internal memory.

[0161] In a possible implementation, determining the second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0162] Determining a first data amount corresponding to the group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels;

[0163] Determine the initial size of the input data according to the initial size of the output data;

[0164] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0165] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0166] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

[0167] In a possible implementation, determining the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0168] According to the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the groups is determined.

[0169] In a possible implementation, segmenting the data in the external memory includes:

[0170] The data in the external memory is rearranged, and the rearranged data is segmented.

[0171] In a possible implementation manner, the data includes characteristic data, and the rearranging of the data in the external memory includes:

[0172] Sorting the feature data according to the input order, and grouping the sorted feature data according to a first set number;

[0173] In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

[0174] In a possible implementation, the data includes convolution kernel data, and the rearranging the data in the external memory includes:

[0175] Sorting the convolution kernel data according to an input order, and grouping the sorted convolution kernel data according to a second set number;

[0176] In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are rearranged according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0177] Since the principle of solving the problem by the computer-readable medium provided above is similar to the data processing method, after the processor executes the computer program in the computer-readable medium above, the steps implemented can refer to the other embodiments above, and the repeated parts will not be repeated.

[0178] Based on the above embodiments, an embodiment of the present application provides a chip, which can implement the following steps:

[0179] Determine a second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernels in the neural network layer;

[0180] Determining a third size of the input data according to the second size of the output data;

[0181] The data in the external memory is segmented according to the third size of the input data and the number of the convolution kernels, and the segmented data is input into the internal memory.

[0182] In a possible implementation, determining the second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0183] Determining a first data amount corresponding to the group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels;

[0184] Determine the initial size of the input data according to the initial size of the output data;

[0185] Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data;

[0186] If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size;

[0187] If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

[0188] In a possible implementation, determining the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes:

[0189] According to the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the groups is determined.

[0190] In a possible implementation, segmenting the data in the external memory includes:

[0191] The data in the external memory is rearranged, and the rearranged data is segmented.

[0192] In a possible implementation manner, the data includes characteristic data, and the rearranging of the data in the external memory includes:

[0193] Sorting the feature data according to the input order, and grouping the sorted feature data according to a first set number;

[0194] In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

[0195] In a possible implementation, the data includes convolution kernel data, and the rearranging the data in the external memory includes:

[0196] Sorting the convolution kernel data according to an input order, and grouping the sorted convolution kernel data according to a second set number;

[0197] In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are rearranged according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

[0198] Since the principle of solving the problem by the chip provided above is similar to the data processing method, the steps implemented by the computer in the above chip can refer to the other embodiments above, and the repeated parts will not be repeated.

[0199] In the specific implementation process, the above-mentioned chip may include a processing circuit (or logic circuit), which implements the steps of any of the above-mentioned data processing methods. Optionally, the above-mentioned chip may also include a storage unit, that is, the memory and the processing circuit may be integrated on the same chip. The storage unit stores a computer program, and the computer program implements the steps of any of the above-mentioned data processing methods when executed by the processing circuit. Optionally, the above-mentioned chip may also include an interface, which may be used to communicate with a module outside the chip, for example, the interface may be used to receive a computer program, and the processing unit and the processing circuit storing the computer program may be respectively arranged on different chips. The specific implementation of various circuits in the chip is not limited in the embodiments of the present application.

[0200] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, chips, or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0201] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernels in the neural network layer; Determining a third size of the input data according to the second size of the output data; According to the third size of the input data and the number of the convolution kernels, the data in the external memory is segmented, and the segmented data is input into the internal memory; Wherein, determining the second size of the output data and the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes: Determining the data volume of the convolution kernel based on the initial number of the convolution kernels and the first size of the convolution kernel; If the sum of the initial size of the output data, the determined size of the input data and the amount of data of the convolution kernel does not exceed the storage space of the memory, the initial number of convolution kernels is determined as the number of convolution kernels, and the initial size of the output data is determined as the second size of the output data.

2. The method according to claim 1, characterized in that The determining, according to the storage space of the memory and the first size of the convolution kernel in the neural network layer, the second size of the output data and the number of the convolution kernels comprises: Determining a first data amount corresponding to the group of convolution kernels according to a first size of the convolution kernel, a first number of channels of the convolution kernel, and a number of a group of convolution kernels; Determine the initial size of the input data according to the initial size of the output data; Determining a second data amount corresponding to one calculation according to the first data amount, the initial size of the output data, and the initial size of the input data; If the second data amount exceeds the storage space of the memory, and / or if the initial size of the input data is greater than or equal to the set size, adjusting the initial size of the output data until the adjusted second data amount is less than or equal to the storage space of the memory, and the adjusted initial size of the input data is less than the set size; If the second data amount exceeds the storage space of the memory and the initial size of the input data is smaller than the set size, the initial size of the output data is determined as the second size of the output data.

3. The method according to claim 2, characterized in that The determining the number of convolution kernels according to the storage space of the memory and the first size of the convolution kernel in the neural network layer includes: According to the storage space of the memory, the second size of the output data and the first data amount corresponding to the group of convolution kernels, the number of groups of convolution kernels is determined, and the number of convolution kernels in the groups is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The segmenting of the data in the external memory includes: The data in the external memory is rearranged, and the rearranged data is segmented.

5. The method according to claim 4, characterized in that The data includes characteristic data, and the rearranging of the data in the external memory includes: Sorting the feature data according to the input order, and grouping the sorted feature data according to a first set number; In each set of feature data, the pixels in the feature data in the set of feature data are rearranged according to the input order of the feature data and the arrangement order of the pixels.

6. The method according to claim 4, characterized in that The data includes convolution kernel data, and the rearranging of the data in the external memory includes: Sorting the convolution kernel data according to an input order, and grouping the sorted convolution kernel data according to a second set number; In each group of convolution kernel data, the pixels in the convolution kernel data in the group of convolution kernel data are rearranged according to the input order of the convolution kernel data, the arrangement order of the channels, and the arrangement order of the pixels.

7. A data processing device, characterized in that: The device comprises: A first determination module, used to determine a second size of output data and the number of convolution kernels according to a storage space of the memory and a first size of the convolution kernel in the neural network layer; A second determining module, configured to determine a third size of the input data according to the second size of the output data; A segmentation module, used for segmenting the data in the external memory according to the third size of the input data and the number of the convolution kernels, and inputting the segmented data into the internal memory; The first determination module is specifically used to determine the data volume of the convolution kernel based on the initial number of the convolution kernels and the first size of the convolution kernel; if the sum of the initial size of the output data, the determined size of the input data and the data volume of the convolution kernel does not exceed the storage space of the memory, then the initial number of the convolution kernels is determined as the number of convolution kernels, and the initial size of the output data is determined as the second size of the output data.

8. A chip, characterized in that: The chip is used to execute the steps of the data processing method in any one of claims 1-6.

9. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the data processing method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 6.

Images