Image data processing method, neural network processor, chip and electronic device

Abstract

The application discloses an image data processing method, a neural network processor, a chip and an electronic device. The method comprises the following steps: a second calculation unit performs feature reconstruction on an n-th frame of image, wherein n is an integer, and n is greater than or equal to 2; reference data of the feature reconstruction is based on intermediate data of an (n-1)-th frame of image in cache; and the intermediate data of the (n-1)-th frame of image is obtained after the (n-1)-th frame of image is subjected to feature extraction by a first calculation unit. The application can improve the inference efficiency of an AI chip on a neural network.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular relates to an image data processing method, a neural network processor, a chip, and an electronic device. Background Art

[0002] With the development of electronic device technology, AI (Artificial Intelligence) chips have been introduced. These chips can be used to implement neural network reasoning. In related technologies, during the neural network reasoning process, the computing units of AI chips remain in a waiting state for a long time, resulting in wasted computing units and increased system latency, which in turn reduces the AI ​​chip's neural network reasoning efficiency. Summary of the Invention

[0003] The embodiments of the present application provide an image data processing method, a neural network processor, a chip, and an electronic device, which can improve the efficiency of AI chip reasoning on neural networks.

[0004] In a first aspect, an embodiment of the present application provides an image data processing method, the method comprising:

[0005] The second computing unit performs feature reconstructing on the n-th frame image, where n is an integer and n≥2, and the reference data for the feature reconstruction is based on the cached intermediate data of the n-1-th frame image, and the intermediate data of the n-1-th frame image is obtained after the feature extraction of the n-1-th frame image by the first computing unit.

[0006] In a second aspect, an embodiment of the present application provides a neural network processor, comprising:

[0007] A first computing unit is configured to perform feature extraction on the image;

[0008] a second computing unit configured to perform feature reconstruction on the image, wherein reference data for the feature reconstruction is based on intermediate data of a cached previous frame of image, wherein the intermediate data is obtained after feature extraction of the previous frame of image by the first computing unit;

[0009] The memory is configured to cache the intermediate data after the feature extraction.

[0010] In a third aspect, an embodiment of the present application provides a chip, including the neural network processor provided in an embodiment of the present application.

[0011] In a fourth aspect, an embodiment of the present application also provides an electronic device, comprising a memory, a processor, and a chip. The processor is used to execute the image data processing method provided by the embodiment of the present application by calling a computer program stored in the memory, and the chip is the chip provided by the embodiment of the present application.

[0012] In an embodiment of the present application, when the second computing unit performs feature reconstruction on the nth frame image, the reference data for feature reconstruction is based on the intermediate data of the cached n-1th frame image, and the intermediate data of the n-1th frame image is obtained after the feature extraction of the n-1th frame image by the first computing unit, that is, when the second computing unit performs feature reconstruction on the nth frame image, it can refer to the intermediate data generated after the feature extraction of the cached n-1th frame image. In addition, since the neural network reasoning process is divided into two halves, feature extraction and feature reconstruction, the feature extraction is calculated by the first computing unit, and the feature reconstruction is calculated by the second computing unit, the calculation of the second computing unit uses the intermediate result cached after the calculation of the first computing unit, which avoids unnecessary waiting time when the second computing unit calculates and improves the parallelism of the system. Therefore, the embodiment of the present application can improve the reasoning efficiency of the AI ​​chip for neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and beneficial effects of the present application apparent.

[0014] Figure 1 It is a flowchart of the image data processing method provided in an embodiment of the present application.

[0015] Figure 2 This is a computational diagram of the Unet convolutional neural network performing frame-by-frame sequential reasoning in related technologies.

[0016] Figure 3 This is another flowchart of the image data processing method provided in an embodiment of the present application.

[0017] Figure 4 This is a scenario diagram of the image data processing method provided in an embodiment of the present application.

[0018] Figure 5 This is another scenario diagram of the image data processing method provided in an embodiment of the present application.

[0019] Figure 6 It is a structural diagram of the Unet convolutional neural network provided in the embodiment of the present application.

[0020] Figure 7 Schematic diagram of the structure of the neural network processor provided in the embodiment of the present application.

[0021] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application.

[0022] Figure 9 This is another structural diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Please refer to the accompanying drawings, in which the same reference numerals represent the same components. The principles of this application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated embodiments of this application and should not be construed as limiting other embodiments of this application that are not described in detail herein.

[0024] It is understandable that the execution subject of the embodiments of the present application can be an electronic device such as a smart phone, a PC (Personal Computer), a tablet computer, etc.

[0025] An electronic device may be provided with an AI chip, which may also be referred to as an AI accelerator or computing card. It is a chip specifically used to process a large number of computing tasks in artificial intelligence applications. The AI ​​chip can process data such as images, voice, and video. Optionally, the AI ​​chip includes an NPU (Neural network processing unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a brain-like chip, a reconfigurable general-purpose AI chip, and the like. The technical solutions provided in the embodiments of the present application can be executed by an AI chip.

[0026] See also Figure 1 , Figure 1 : is a flow chart of the image data processing method provided in an embodiment of the present application. The process of the image data processing method may include:

[0027] 101. The second computing unit performs feature reconstruction on the n-th frame image, where n is an integer and n≥2, and the reference data for feature reconstruction is based on the cached intermediate data of the n-1-th frame image, and the intermediate data of the n-1-th frame image is obtained after the feature extraction of the n-1-th frame image by the first computing unit.

[0028] With the development of electronic device technology, AI (Artificial Intelligence) chips have been introduced. These chips can be used to implement neural network reasoning. In related technologies, AI chips typically use a layer-by-layer sequential reasoning approach to implement neural network reasoning. First, they calculate the first half of feature extraction, then the second half of feature reconstruction. If the data from the previous layer has not yet reached the computing unit, the computing unit remains in a waiting state.

[0029] See also Figure 2 , Figure 2 This is a computational diagram of the Unet convolutional neural network (CNN) used in related art for frame-by-frame sequential reasoning. An image is input through the input layer. The input layer's feature extraction unit performs a convolution operation on the input image to obtain a feature map for the input layer. Multiple convolutional layers may follow the input layer's feature extraction unit. The number of convolutional layers depends on the structure of the Unet convolutional neural network, with each convolutional layer equipped with a feature extraction unit. After obtaining the input layer's feature map, the input layer's feature extraction unit feeds it into the first feature fusion layer, then feeds it into the next convolutional layer. The feature extraction unit in the next convolutional layer then performs a convolution operation on the input layer's feature map to obtain the next feature map. The feature extraction unit in the next convolutional layer then feeds the next feature map into the first feature fusion layer, then feeds it into the next convolutional layer, and so on. The first feature fusion layer fuses the input layer's feature map with the next feature map. It can be understood that the first feature fusion layer fuses the convolutional feature maps of the input layer and each convolutional layer, ultimately obtaining the fused first feature map.

[0030] The first feature after feature fusion Figure 1 One path passes through the first 1×1 convolution layer for convolution operation, the other path passes through the second 1×1 convolution layer for convolution operation, and passes through the first feature extraction layer for feature extraction. The feature maps obtained by the two paths are fused through the second feature fusion layer to obtain the fused second feature map.

[0031] The fused second feature map is extracted by the second feature extraction layer and then input into the feature reconstruction layer for feature reconstruction. The resulting image after the feature reconstruction layer can be divided into two cases: 1. If the input image is noisy, the output is a denoised image; 2. If the input image is normal, the image is segmented to obtain a segmented image. This means that features are extracted from the portion of interest in the input image to obtain the desired image.

[0032] The inference process first reads the data that the first half of the network relies on from double data rate synchronous dynamic random access memory (DDR SDRAM), dynamic random access memory (DRAM), or static random access memory (SRAM) into on-chip memory (OCM) and performs calculations. The second half of the calculation depends on the results of the first half for subsequent calculations. The data reading and calculation in the first half are usually accompanied by input and output (IO) blocking, causing system delays.

[0033] For video denoising neural networks, frame-by-frame inference is performed during the inference process. The feature data generated by the inference of the previous frame of image is discarded after the inference is completed and is not retained. When calculating the next frame of image, the feature data of the previous frame of image is not fully utilized, resulting in repeated calculations, which in turn causes additional power consumption.

[0034] From the above, we can see that since the computing unit is in a waiting state for a long time, it causes waste of computing units and increases system delays, thereby reducing the AI ​​chip's reasoning efficiency on neural networks.

[0035] For example, in an embodiment of the present application, a Unet convolutional neural network acceleration method based on a ping-pong computing mode can be used to divide the reasoning process of the neural network into feature extraction and feature reconstruction. Feature extraction is the first half of the reasoning process, and feature reconstruction is the second half of the reasoning process. The entire process or part of the process of feature extraction can be calculated using a first computing unit, and the entire process or part of the process of feature reconstruction can be calculated using a second computing unit, that is, feature extraction and feature reconstruction are calculated in parallel using different computing units. During the calculation process of feature reconstruction, the intermediate result of the previous frame image after feature extraction can be referenced to accelerate the neural network reasoning process.

[0036] It should be noted that in practical applications, the reasoning process of the neural network can be divided into multiple segments according to actual needs, and each segment is calculated by a different computing unit to achieve the effect of parallel computing.

[0037] Specifically, the reasoning process of the neural network is divided into feature extraction and feature reconstruction. Feature extraction and feature reconstruction are respectively calculated in parallel by different computing units. For example, feature extraction is calculated by a first computing unit, and feature reconstruction is calculated by a second computing unit. The first computing unit and the second computing unit can be calculated in parallel. In one embodiment, the first computing unit and the second computing unit can be set in the on-chip memory of the chip, such as the on-chip memory of the AI ​​chip.

[0038] In an embodiment of the present application, after the nth frame image is input through the input layer, the first computing unit can be used to extract features of the nth frame image, and the second computing unit can be used to reconstruct features of the nth frame image, where n is an integer and n≥2.

[0039] For example, when reconstructing features for the nth frame, reference may be made to the cached intermediate data for the n-1th frame, which is obtained after feature extraction of the n-1th frame by the first computing unit. When extracting features from the n-1th frame, the first computing unit may perform convolution operations on the n-1th frame through multiple convolutional layers.

[0040] Specifically, when the first computing unit extracts features from the n-1th frame image and obtains feature data, the intermediate data in the feature data that can be used for reference in reconstructing the features of the nth frame image is cached, that is, the intermediate data of the n-1th frame image is cached. For example, it can be cached in a memory, which can be a double-rate synchronous dynamic random access memory, a dynamic random access memory or a static random access memory, etc. The embodiment of the present application does not limit the type of cached memory.

[0041] When the second computing unit performs feature reconstruction on the nth frame, the cached intermediate data of the n-1th frame can be read into the second computing unit as reference data for the second computing unit's feature reconstruction of the nth frame. Thus, when reconstructing features on an image, reference can be made to the intermediate data after feature extraction from the previous frame. This reduces unnecessary waiting time for the second computing unit and accelerates the neural network inference process.

[0042] It can be understood that in the embodiment of the present application, when the second computing unit performs feature reconstruction on the nth frame image, the reference data for feature reconstruction is based on the intermediate data of the cached n-1th frame image, and the intermediate data of the n-1th frame image is obtained after the feature extraction of the n-1th frame image by the first computing unit, that is, when the second computing unit performs feature reconstruction on the nth frame image, it can refer to the intermediate data generated after the feature extraction of the cached n-1th frame image. In addition, since the process of neural network reasoning is divided into two halves, feature extraction and feature reconstruction, the feature extraction is calculated by the first computing unit, and the feature reconstruction is calculated by the second computing unit, the calculation of the second computing unit uses the intermediate result cached after the calculation of the first computing unit, which avoids unnecessary waiting time when the second computing unit calculates and improves the parallelism of the system. Therefore, the embodiment of the present application can improve the efficiency of AI chip reasoning on neural networks.

[0043] See also Figure 3 , Figure 3FIG. 1 is another flow chart of the image data processing method provided in an embodiment of the present application. The flow of the image data processing method may include:

[0044] 201. A first computing unit extracts features from an n-th frame image to obtain second feature data.

[0045] For example, in an embodiment of the present application, when the input layer inputs the nth frame image, that is, when the nth frame image arrives, the nth frame image is read into the first computing unit so that the first computing unit can extract features from the nth frame image, where n is an integer and n≥2. The feature extraction process is the same as the feature extraction process for the n-1th frame image, and will not be repeated here. After feature extraction, the second feature data can be obtained. The first feature data is the feature map.

[0046] For example, in one embodiment, the first computing unit in step 201 extracts features from the n-th frame image to obtain second feature data, which may include:

[0047] The first computing unit obtains the second feature data after performing a multi-layer convolution operation on the n-th frame image.

[0048] For example, see Figure 4 , Figure 4 This is a schematic diagram of the segmented reasoning scenario provided by the embodiment of the present application. The reasoning process of the entire neural network is divided into feature extraction and feature reconstruction, wherein feature extraction is calculated by the first computing unit and feature reconstruction is calculated by the second computing unit. Feature extraction is Figure 4 The feature extraction process performed by the feature extraction module in Figure 4 The feature reconstruction process performed by the feature reconstruction module in . That is, each frame of image is divided into two halves during calculation: feature extraction and feature reconstruction. They are executed in sequence. Feature extraction is completed by the first calculation unit, and feature reconstruction is completed by the second calculation unit.

[0049] It should be noted that when the first computing unit extracts features from the n-th frame image, Figure 4The feature extraction unit in each convolutional layer performs feature extraction. When the feature extraction unit in each convolutional layer performs feature extraction, feature extraction is performed by convolution operation. For example, when the input layer inputs the nth frame image, the feature extraction unit in the input layer performs convolution operation on the nth frame image to realize feature extraction of the nth frame image. After feature extraction, a feature map is obtained, and the feature map is input into the next convolutional layer, and convolution operation is performed on it by the feature extraction unit of the next convolutional layer to realize feature extraction again, and so on, until it is input into the last convolutional layer for feature extraction. It should be noted that the feature map of each convolutional layer after convolution operation will be input into the first feature fusion layer.

[0050] 202. The first calculation unit obtains intermediate data of the n-th frame image from the second feature data, and the intermediate data of the n-th frame image is used as reference data for feature reconstruction of the (n+1)-th frame image.

[0051] For example, in an embodiment of the present application, when the first computing unit extracts features from the nth frame image and obtains second feature data, not all of the second feature data can be used as reference data for feature reconstruction of the n+1th frame image. Only a portion of the second feature data can be used as reference data for feature reconstruction of the n+1th frame image. That is, this portion of reference data will be used for feature reconstruction of the n+1th frame image, and this portion of reference data will be used as intermediate data. The reference data used for feature reconstruction of the n+1th frame image, that is, the intermediate data of the nth frame image, can be extracted from the first feature data.

[0052] It should be noted that the intermediate data may be a plurality of feature maps obtained after the image undergoes multi-layer convolution operations in the first computing unit.

[0053] 203. Cache the intermediate data of the n-th frame image.

[0054] For example, in an embodiment of the present application, after obtaining the intermediate data of the n-th frame image from the second feature data, that is, obtaining the reference data used for reconstructing the features of the n+1-th frame image, the intermediate data of the n-th frame image can be cached, that is, the intermediate data of the n-th frame image calculated by the first calculation unit can be cached for reference when reconstructing the features of the n+1-th frame image. It should be noted that when caching, it can be cached in a memory, which can be DDRSDRAM, DRAM or SRAM, etc. Figure 4 It can also be seen from the scene diagram that the intermediate data calculated from the previous frame image will be cached in the memory so that it can be used as a reference when reconstructing the features of the next frame image.

[0055] It should be noted that when the second computing unit reconstructs features of the nth frame image, the first computing unit simultaneously extracts features of the (n+1)th frame image. Therefore, the first computing unit and the second computing unit can realize parallel computing.

[0056] In one embodiment, when the second computing unit performs feature reconstruction on the nth frame image, the first computing unit simultaneously performs feature extraction on the (n+1)th frame image, which may include:

[0057] Reading the cached intermediate data of the (n-1)th frame of image into the second computing unit;

[0058] The second calculation unit reconstructs features of the n-th frame image based on the intermediate data of the n-1-th frame image, and at the same time, the first calculation unit extracts features of the n+1-th frame image.

[0059] For example, in an embodiment of the present application, when the nth frame image arrives, that is, the nth frame image is input through the input layer, the nth frame image can be obtained at this time. After obtaining the nth frame image, the first computing unit is used to extract features from the nth frame image. It can be understood that after the feature extraction of the nth frame image, the intermediate data of the feature data generated by the feature extraction that can be used for feature reconstruction of the n+1th frame image is cached in the memory for reference when reconstructing the features of the n+1th frame image.

[0060] Furthermore, when the second computing unit reconstructs features for the nth image frame, it references the intermediate data for the n-1th image frame. Since the intermediate data for the n-1th image frame is cached in memory, it is necessary to read the cached intermediate data for the n-1th image frame into the second computing unit, which then reconstructs features for the nth image frame based on the intermediate data for the n-1th image frame. During the feature reconstruction process for the nth image frame, the second computing unit references the intermediate data for the n-1th image frame.

[0061] It should be noted that while the second computing unit is reconstructing features on the nth frame, the first computing unit can simultaneously extract features from the incoming n+1th frame. That is, when the second computing unit is reconstructing features on the nth frame, the n+1th frame arrives, and the first computing unit will then extract features from the acquired n+1th frame. Thus, while the second computing unit is reconstructing features on the nth frame, the first computing unit will extract features from the acquired n+1th frame, enabling parallel computations between the first and second computing units.

[0062] In an embodiment of the present application, when the second computing unit calculates, after feature reconstruction of the nth frame image is performed based on the cached intermediate data of the previous frame image calculated by the first computing unit, image data generated by feature reconstruction can be obtained, and the second computing unit can output the image data after feature reconstruction of the nth frame image.

[0063] It should be noted that after executing the process of 203 , the process of 204 or 206 can be executed.

[0064] 204. The second computing unit performs multiple feature fusion and feature extraction on the intermediate data of the (n-1)th frame image to obtain first feature data.

[0065] 205. The second computing unit performs feature reconstruction on the first feature data.

[0066] For example, see Figure 4 , the first feature fusion layer will fuse the feature map of the input layer with the next feature map. It can be understood that the first feature fusion layer will fuse the feature maps of the input layer and each convolution layer after convolution operation, and finally obtain the first feature map after feature fusion. Figure 1 One path passes through the first 1×1 convolutional layer for convolution, the other path passes through the second 1×1 convolutional layer for convolution, and then passes through the first feature extraction layer for feature extraction. The feature maps obtained from the two paths are fused through the second feature fusion layer to obtain a second feature map after feature fusion. The second feature map after feature fusion is extracted through the second feature extraction layer and then input into the two feature reconstruction layers for feature reconstruction.

[0067] It should be noted that when performing feature fusion and feature extraction, multiple feature fusion and feature extraction can be performed to obtain the first feature data, and the size of the convolution kernel of the convolution layer can be adjusted accordingly according to specific needs. After multiple feature fusions and feature extractions, the first feature data obtained can be reconstructed through multiple feature reconstruction layers. The number of feature reconstruction layers can be adjusted accordingly according to specific needs. The embodiment of the present application does not limit the number of feature reconstruction layers.

[0068] It can be understood that after feature extraction is performed on the previous frame image, the intermediate results that can be used for the current frame image are cached in DDR SDRAM, DRAM or SRAM. When the current frame image arrives, the first computing unit extracts features from the current frame image, and after the current frame image is calculated by the first computing unit, the intermediate results that can be used for reference by the next frame image are cached in DDR SDRAM, DRAM or SRAM. Similarly, when the mth frame image arrives, the second computing unit reconstructs the features of the mth frame image, and refers to the intermediate results of the m-1th frame image after feature extraction.

[0069] When the second computing unit uses the intermediate results of the previous frame to reconstruct the features of the current frame, it can achieve the purpose of accelerated or parallel computing. In the related art, due to the lack of cache, the first computing unit extracts the features of each frame as it arrives. After the first computing unit completes the calculation, the second computing unit begins to calculate and reconstruct the current frame, which will cause system delays.

[0070] 206. The first computing unit performs feature extraction on the (n+1)th frame image.

[0071] For example, while the second computing unit is reconstructing features on the nth frame image, the first computing unit can simultaneously extract features on the n+1th frame image. The process of extracting features on the n+1th frame image is similar to the process of extracting features on the n-1th frame image or the nth frame image, and will not be repeated here.

[0072] It can be understood that in the embodiment of the present application, in the reasoning process of the neural network, a layer-by-layer reasoning method is adopted. When the data of the previous convolutional layer has not reached the DDR SDRAM, DRAM or SRAM, the second computing unit can perform calculations to reduce the time that the second computing unit is in a waiting state. For the video denoising neural network, frame-by-frame reasoning is performed during the reasoning process. The intermediate data in the feature data generated by the feature extraction of the previous frame image is cached after the feature extraction is completed, and the intermediate data of the previous frame image is fully utilized when calculating the next frame image (i.e., feature reconstruction). When the second computing unit reconstructs the features of the current frame image, the first computing unit simultaneously extracts the features of the next frame image to achieve the purpose of parallel computing. It can be seen that the reasoning process adopts a parallel method of multiple computing units, and each core processes data of different layers in parallel to achieve the purpose of parallelism and acceleration, thereby reducing the blocking time caused by data reading and writing.

[0073] See also Figure 5 , Figure 5 This is another scenario diagram of the image data processing method provided by an embodiment of the present application. When a frame of image is input to the input layer, the frame of image can be stored in DDR SDRAM, DRAM or SRAM. When the first computing unit needs to calculate, the frame of image is read into the first computing unit. After the first computing unit performs calculation, the intermediate data of the feature data obtained after the calculation is stored in DDR SDRAM, DRAM or SRAM. When the second computing unit needs to calculate the frame of image, the intermediate data of the previous frame of image is read from DDR SDRAM, DRAM or SRAM. After the second computing unit completes the calculation of the frame of image, the calculation result is output through the output layer.

[0074] For example, see Figure 6 , Figure 6 This is a schematic diagram of the structure of the Unet convolutional neural network provided in an embodiment of the present application. The Unet convolutional neural network includes convolution operations, upsampling operations, maximum pooling operations, stacking operations, etc. Convolution operations refer to operations that process the original image according to a template to obtain a new image; upsampling operations refer to operations that amplify the original image so that the amplified image can be displayed on a higher resolution display device; maximum pooling operations refer to operations that use the maximum value in an image area as the pooled value of the area; and stacking operations refer to operations that connect two units together.

[0075] The mth layer in the embodiment of the present application is Figure 6 The input layer where the solid line box is located, the nth layer in the embodiment of the present application is Figure 6 At the output layer located at the dotted box in the figure, the feature map of layer m needs to be concatenated (i.e., stacked or fused) with the feature map of layer n. Before the concatenation, the feature map of layer m is generally stored in DDR SDRAM, DRAM, or SRAM. This data is used for calculations during the concatenation and is temporarily unused during inference at the intermediate level. The feature map of layer m is not read from DDR SDRAM, DRAM, or SRAM until the second computing unit performs calculations. For example, when calculating the nth layer of the neural network, the feature map of layer n needs to be stacked with the feature map of layer m. Therefore, layer n needs to refer to the feature map of layer m for calculation. At this time, the feature map of layer m is read from DDR SDRAM, DRAM, or SRAM.

[0076] If the computation follows the related art approach of first extracting features and then reconstructing them, the computing unit remains in a waiting state, wasting computing time and increasing system latency. In particular, during the feature extraction process, the input image data must be read from the DDR SDRAM and cached until a sufficient amount has been reached before the computing unit can begin calculations. This results in a prolonged waiting period and waste of computing units.

[0077] Through the scheme in the embodiment of the present application, the reasoning process of the neural network is divided into feature extraction and feature reconstruction, and the calculations of feature extraction and feature reconstruction are performed separately, that is, feature extraction and feature reconstruction are calculated separately, and different calculation units are used for calculation respectively. Feature extraction is calculated by the first calculation unit, and feature reconstruction is calculated by the second calculation unit. When the second calculation unit performs calculations, the intermediate result of the previous frame image after feature extraction by the first calculation unit is reused. By using the intermediate result of the feature extraction cache for calculation of feature reconstruction, unnecessary waiting time when the second calculation unit calculates is avoided, and the parallelism of the system is improved. By reducing repeated calculations through data reuse, the purpose of reducing the amount of calculation and reducing power consumption is achieved.

[0078] At present, mainstream neural networks, such as Unet and face recognition neural networks, are all based on the encoder / decoder architecture, which can split the neural network reasoning process into feature extraction and feature reconstruction. And feature reconstruction can be calculated based on the intermediate data of the previous frame image cached during feature extraction. Therefore, the solution of the embodiment of the present application can be more generally applicable to the deployment and acceleration of various neural network scenarios.

[0079] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of the neural network processor provided in the embodiment of the present application. The data processing device 300 may include: a first computing unit 301, a second computing unit 302, and a memory 303.

[0080] A first computing unit 301 is configured to perform feature extraction on an image;

[0081] The second computing unit 302 is configured to perform feature reconstruction on the image, wherein reference data for the feature reconstruction is based on intermediate data of a cached previous frame of image, wherein the intermediate data is obtained after feature extraction of the previous frame of image by the first computing unit;

[0082] The memory 303 is configured to cache the intermediate data after the feature extraction.

[0083] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the process in the image data processing method provided in this embodiment.

[0084] An embodiment of the present application provides a chip, which includes the neural network processor provided in an embodiment of the present application.

[0085] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a chip. The processor is used to execute the process in the image data processing method provided by this embodiment by calling the computer program stored in the memory. The chip is the chip provided by the embodiment of the present application.

[0086] For example, the electronic device may be a mobile terminal such as a tablet computer, a PC, or a smart phone. Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0087] The electronic device 400 may include components such as a memory 401, a processor 402, and a chip 403. Those skilled in the art will appreciate that Figure 8 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0088] Memory 401 can be used to store applications and data. The applications stored in memory 401 include executable code. Applications can be composed of various functional modules. Processor 402 executes various functional applications and data processing by running the applications stored in memory 401.

[0089] The processor 402 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing applications stored in the memory 401 and calling data stored in the memory 401, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0090] Chip 403 can be used to handle a large number of computing tasks in artificial intelligence applications, such as processing image data.

[0091] In this embodiment, the processor 402 in the electronic device loads the executable code corresponding to one or more application processes into the memory 401 according to the following instructions, and the processor 402 runs the application stored in the memory 401 to execute:

[0092] The second computing unit performs feature reconstructing on the n-th frame image, where n is an integer and n≥2, and the reference data for the feature reconstruction is based on the cached intermediate data of the n-1-th frame image, and the intermediate data of the n-1-th frame image is obtained after the feature extraction of the n-1-th frame image by the first computing unit.

[0093] See also Figure 9, the electronic device 400 may include components such as a memory 401 , a processor 402 , a chip 403 , a battery 404 , a microphone 405 , and a speaker 406 .

[0094] Memory 401 can be used to store applications and data. The applications stored in memory 401 include executable code. Applications can be composed of various functional modules. Processor 402 executes various functional applications and data processing by running the applications stored in memory 401.

[0095] The processor 402 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing applications stored in the memory 401 and calling data stored in the memory 401, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0096] Chip 403 can be used to handle a large number of computing tasks in artificial intelligence applications, such as processing image data.

[0097] The battery 404 may be used to provide power support to various components of the electronic device, thereby ensuring the normal operation of each component.

[0098] The microphone 405 may be used to receive sound signals in the surrounding environment, for example, may be used to receive voice uttered by a user.

[0099] The speaker 406 can be used to play sound signals.

[0100] In this embodiment, the processor 402 in the electronic device loads the executable code corresponding to one or more application processes into the memory 401 according to the following instructions, and the processor 402 runs the application stored in the memory 401 to execute:

[0101] The second computing unit performs feature reconstructing on the n-th frame image, where n is an integer and n≥2, and the reference data for the feature reconstruction is based on the cached intermediate data of the n-1-th frame image, and the intermediate data of the n-1-th frame image is obtained after the feature extraction of the n-1-th frame image by the first computing unit.

[0102] In one embodiment, when the second computing unit performs feature reconstruction on the nth frame image, the processor 402 may execute: the second computing unit performs multiple feature fusion and feature extraction on the intermediate data of the n-1th frame image to obtain first feature data; and the second computing unit performs feature reconstruction on the first feature data.

[0103] In one embodiment, when the second computing unit performs feature reconstruction on the nth frame image, the first computing unit simultaneously performs feature extraction on the (n+1)th frame image.

[0104] In one embodiment, when the processor 402 executes the second computing unit to reconstruct features of the nth frame image, and the first computing unit simultaneously extracts features of the n+1th frame image, it can execute: reading the cached intermediate data of the n-1th frame image to the second computing unit; the second computing unit reconstructs features of the nth frame image based on the intermediate data of the n-1th frame image, and at the same time, the first computing unit extracts features of the n+1th frame image.

[0105] In one embodiment, before the second computing unit performs feature reconstruction on the n-th frame image, the processor 402 may execute: the first computing unit performs feature extraction on the n-th frame image to obtain second feature data.

[0106] In one embodiment, when the processor 402 executes the first computing unit to extract features from the nth frame image and obtains second feature data, it can execute: the first computing unit performs multi-layer convolution operations on the nth frame image to obtain the second feature data.

[0107] In one embodiment, the processor 402 can also execute: the first computing unit obtains the intermediate data of the nth frame image from the second feature data, and the intermediate data of the nth frame image is used as reference data when reconstructing the features of the n+1th frame image; and caches the intermediate data of the nth frame image.

[0108] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above and will not be repeated here.

[0109] The data processing device provided in the embodiment of the present application belongs to the same concept as the image data processing method in the above embodiment. Any method provided in the image data processing method embodiment can be run on the neural network processor. The specific implementation process is detailed in the data processing method embodiment and will not be repeated here.

[0110] It should be noted that, with respect to the image data processing method described in the embodiments of the present application, persons of ordinary skill in the art will understand that all or part of the process of the image data processing method described in the embodiments of the present application can be implemented by controlling related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, the program can include the process of the embodiment of the image data processing method. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0111] For the neural network processor of the embodiment of the present application, its various functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0112] The above is a detailed introduction to an image data processing method, neural network processor, chip and electronic device provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing image data, characterized in that: The method comprises: The second computing unit performs feature reconstruction on the n-th frame image, wherein n is an integer and n≥2, and the reference data for the feature reconstruction is based on the intermediate data of the cached n-1-th frame image, and the intermediate data of the n-1-th frame image is obtained after the n-1-th frame image is feature extracted by the first computing unit; wherein the feature extraction is calculated by the first computing unit, and the feature reconstruction is calculated by the second computing unit, and the first computing unit and the second computing unit calculate in parallel.

2. The image data processing method according to claim 1, wherein: The second computing unit performs feature reconstruction on the n-th frame image, including: The second computing unit performs multiple feature fusion and feature extraction on the intermediate data of the (n-1)th frame of image to obtain first feature data; The second calculation unit performs feature reconstruction on the first feature data.

3. The image data processing method according to claim 1, wherein: When the second computing unit performs feature reconstruction on the nth frame image, the first computing unit simultaneously performs feature extraction on the (n+1)th frame image.

4. The image data processing method according to claim 2, wherein: When the second computing unit performs feature reconstruction on the nth frame image, the first computing unit simultaneously performs feature extraction on the (n+1)th frame image, including: Reading the cached intermediate data of the (n-1)th frame of image into the second computing unit; The second calculation unit reconstructs features of the n-th frame image based on the intermediate data of the n-1-th frame image, and at the same time, the first calculation unit extracts features of the n+1-th frame image.

5. The image data processing method according to claim 1, wherein: Before the second computing unit performs feature reconstruction on the nth frame image, the method further includes: The first calculation unit performs feature extraction on the n-th frame image to obtain second feature data.

6. The image data processing method according to claim 5, characterized in that: The first computing unit extracts features from the n-th frame image to obtain second feature data, including: The first computing unit obtains the second feature data after performing a multi-layer convolution operation on the n-th frame image.

7. The image data processing method according to claim 5 or 6, characterized in that: The method further comprises: The first calculation unit obtains intermediate data of the n-th frame image from the second feature data, and the intermediate data of the n-th frame image is used as reference data when reconstructing features of the (n+1)-th frame image; The intermediate data of the n-th frame image is cached.

8. A neural network processor, characterized in that include: A first computing unit is configured to perform feature extraction on the image; a second computing unit configured to perform feature reconstruction on the image, wherein reference data for the feature reconstruction is based on cached intermediate data of a previous frame of image, wherein the intermediate data is obtained after feature extraction of the previous frame of image by the first computing unit; wherein the feature extraction is performed by the first computing unit, and the feature reconstruction is performed by the second computing unit, and the first computing unit and the second computing unit perform the calculations in parallel; The memory is configured to cache the intermediate data after the feature extraction.

9. A chip, characterized in that: Comprising a neural network processor as described in claim 8.

10. An electronic device comprising a memory, a processor and a chip, characterized in that: The processor executes the method according to any one of claims 1 to 7 by calling the computer program stored in the memory, and the chip is the chip according to claim 9.

Images