Convolution engine, method for processing data and electronic device

Abstract

The present disclosure relates to a configurable convolution engine for interleaved channel data, and provides the following embodiments, which relate to a configurable convolution engine that receives configuration information to perform convolution and other deep machine learning operations on stream input data of various formats. The convolution engine may include two convolution circuits, each of which generates a stream of values ​​by applying a convolution kernel to input data. The streams of values ​​may each define one or more image data channels. A channel merging circuit combines the streams of values ​​from each convolution circuit according to a selected operating mode. In one mode, a first stream and a second stream from the convolution circuit are merged into an output stream having a combined channel of the first stream and the second stream in an interleaved manner. In another mode, the first stream from the first convolution circuit is fed into the input of the second convolution circuit.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of a patent application with an international application date of March 6, 2018, an international application number of PCT / US2018 / 021210, an application number of 201880027358.X, and a name of “Configurable Convolution Engine for Interleaved Channel Data”, which entered the Chinese national phase on October 24, 2019. Background Art

[0003] Image data captured by an image sensor or received from other data sources is typically processed in an image processing pipeline before further processing or consumption. For example, the raw image data may be corrected, filtered, or otherwise modified before being provided to subsequent components such as a video encoder. In order to perform correction or enhancement on the captured image data, various components, unit levels, or modules may be employed.

[0004] Such an image processing pipeline can be constructed so that correction or enhancement of captured image data can be performed in an advantageous manner without consuming other system resources. Although many image processing algorithms can be performed by executing software programs on a central processing unit (CPU), executing these programs on the CPU will consume a large amount of bandwidth of the CPU and other peripheral resources as well as increase power consumption. Therefore, the image processing pipeline is usually implemented as a hardware component separate from the CPU and dedicated to executing one or more image processing algorithms.

[0005] Various types of image processing involve convolution between a kernel and data. Different kernels can be used, for example, to blur, sharpen, emboss, or perform edge detection in an image. Such convolution operations are usually performed by the CPU, which reduces the availability of the CPU for other processes. Summary of the invention

[0006] The embodiment relates to a configurable convolution engine, which is used to perform convolution and machine learning operations on input data of various channels in a desired manner by configuring the operation of components in the convolution engine. The convolution engine includes a first convolution circuit, a second convolution circuit, and a channel merging circuit coupled to the first convolution circuit and the second convolution circuit. The first convolution circuit and the second convolution circuit each generate a stream of values ​​by applying a convolution kernel to input data, and so on. The stream of values ​​can each define one or more image data channels in an interleaved manner. The channel merging circuit combines the streams of values ​​from the first convolution circuit and the second convolution circuit according to the selected operation mode. In the dual convolution mode, the first stream and the second stream from the convolution circuit are merged into an output stream, which has a combined channel of the first stream and the second stream in an interleaved manner. In the cascade mode, the first stream from the first convolution circuit is fed into the input of the second convolution circuit. The channel merging circuit outputs the result of the second convolution circuit as an output stream. In the parallel mode, the channel merging circuit outputs the first stream and the second stream from the first convolution circuit and the second convolution circuit as separate streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a high-level diagram of an electronic device according to one embodiment.

[0008] Figure 2 is a block diagram showing components in an electronic device according to one embodiment.

[0009] Figure 3 is a block diagram illustrating an image processing pipeline implemented using an image signal processor according to one embodiment.

[0010] Figure 4 is a block diagram illustrating a vision module in an image signal processor according to one embodiment.

[0011] Figure 5 is a block diagram of a convolution engine according to one embodiment.

[0012] Fig. 6A is a block diagram of a dual convolution mode of a convolution engine according to one embodiment.

[0013] Figure 6B is a block diagram of a cascaded mode of a convolution engine according to one embodiment.

[0014] Figure 6C is a block diagram of a parallel mode of a convolution engine according to one embodiment.

[0015] Figure 7 is a flow chart illustrating a method of operating a convolution engine in multiple modes according to one embodiment.

[0016] Figure 8is a block diagram illustrating a convolution core circuit according to one embodiment.

[0017] Fig. 9 is a graph of a nonlinear transformation applied by a response rectifier unit according to one embodiment.

[0018] Fig.10 is a block diagram illustrating a convolution kernel according to one embodiment.

[0019] Fig.11A is a conceptual diagram illustrating the input and output of a multi-plane format convolution core circuit according to one embodiment.

[0020] Fig. 11B is a conceptual diagram illustrating the input and output of a planarized format convolution core circuit according to one embodiment.

[0021] Fig.12 is a block diagram illustrating a spatial combining circuit according to one embodiment.

[0022] Fig.13A and Fig. 13B is a conceptual diagram illustrating the input and output of a multi-plane format spatial binning circuit according to one embodiment.

[0023] Fig. 13C and Fig.13D is a conceptual diagram illustrating the input and output of a planarized format spatial binning circuit according to one embodiment.

[0024] Fig.14 is a flow chart illustrating a method of operating a spatial merging circuit according to one embodiment.

[0025] Fig.15 is a block diagram illustrating a channel merger according to one embodiment.

[0026] Fig.16 is a conceptual diagram illustrating the input and output of a flattened format channel merger according to one embodiment.

[0027] For purposes of illustration only, the drawings depict and the detailed description describe various non-limiting embodiments. DETAILED DESCRIPTION

[0028] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, thereby not unnecessarily obscuring various aspects of the embodiments.

[0029] Embodiments of the present disclosure relate to a configurable convolution engine for performing convolution and per-channel machine learning operations on input data of various channels in a desired manner by configuring the operation of components in the convolution engine. The convolution engine is a circuit comprising a first convolution circuit, a second convolution circuit, and a channel merging circuit coupled to the first convolution circuit and the second convolution circuit. The first convolution circuit and the second convolution circuit each generate a stream of values ​​by applying a convolution kernel to input data, and so on. The streams of values ​​may each define one or more image data channels in an interleaved manner. The channel merging circuit may combine the streams of values ​​from the first convolution circuit and the second convolution circuit according to the selected operating mode. In dual convolution mode, the streams of values ​​are combined into a single output stream having channels from the first stream and channels from the second stream arranged in an interleaved manner.

[0030] Exemplary Electronic Devices

[0031] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a personal digital assistant (PDA) and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Other portable electronic devices, such as wearable devices, laptops, or tablets, may alternatively be used. In some embodiments, the device is not a portable communication device, but rather a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). Figure 1 Example electronic devices are described. Figure 1 (For example, device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0032] Figure 1is a high-level diagram of an electronic device 100 according to one embodiment. The device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. The menu button 104 is used, for example, to navigate to any application in a set of applications executing on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that recognizes a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether the finger on the menu button 104 has a fingerprint that matches a fingerprint stored for unlocking the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

[0033] In some embodiments, the device 100 includes a touch screen 150, a menu button 104, a push button 106 for turning the device on / off and for locking the device, a volume adjustment button 108, a subscriber identity module (SIM) card slot 110, an earphone jack 112, and a docking / charging external port 124. The push button 106 can be used to turn the device on and off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval has passed; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 includes various components, including but not limited to a memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), a peripheral device interface, an RF circuit, an audio circuit, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include Figure 1 Components not shown.

[0034] Device 100 is only one example of an electronic device, and device 100 may have more or fewer components than listed above, some of which may be combined into components or have a different configuration or arrangement. The various components of device 100 listed above are embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits (ASICs).

[0035] Figure 21 is a block diagram illustrating components in device 100 according to one embodiment. Device 100 can perform various operations including image processing. For this and other purposes, device 100 may include image sensor 202, system on chip (SOC) component 204, system memory 230, permanent storage (e.g., flash memory) 228, motion sensor 234, and display 216, among other components. Figure 2 The components shown in are merely illustrative. For example, the device 100 may include Figure 2 Other components not shown in the figure (such as speakers or microphones). In addition, some components (such as motion sensor 234) can be omitted from device 100.

[0036] The image sensor 202 is a component for capturing image data and may be implemented as, for example, a complementary metal oxide semiconductor (CMOS) active pixel sensor, a camera, a camcorder, or other device. The image sensor 202 generates raw image data, which is sent to the SOC component 204 for further processing. In some embodiments, the image data processed by the SOC component 204 is displayed on the display 216, stored in the system memory 230, the permanent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by the image sensor 202 may be a Bayer color filter array (CFA) pattern (hereinafter also referred to as a "Bayer pattern").

[0037] The motion sensor 234 is a component or a group of components for sensing the motion of the device 100. The motion sensor 234 may generate a sensor signal indicating the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various operations, such as turning on the device 100 or rotating the image displayed on the display 216.

[0038] The display 216 is a component for displaying images generated by the SOC component 204. The display 216 may include, for example, a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on the data received from the SOC component 204, the display 116 may display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC component 204, and / or other information received from the user interface of the device 100 (not shown).

[0039] System memory 230 is a component for storing instructions executed by SOC component 204 and for storing data processed by SOC component 204. System memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, system memory 230 can store pixel data or other image data or statistics in various formats.

[0040] Persistent storage 228 is a component for storing data in a non-volatile manner. Persistent storage 228 retains data even if power is unavailable. Persistent storage 228 may be embodied as a read-only memory (ROM), NAND or NOR, flash memory or other non-volatile random access memory device.

[0041] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 may include an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and various other input / output (I / O) interfaces 218 and a bus 232 connecting these subcomponents. The SOC component 204 may include other subcomponents such as Figure 1 More or fewer subcomponents than those shown.

[0042] ISP 206 is hardware that performs various stages of the image processing pipeline. In some embodiments, ISP 206 may receive raw image data from image sensor 202 and process the raw image data into a form usable by other subcomponents of SOC component 204 or components of device 100. ISP 206 may perform various image processing operations, such as image translation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as described below with reference to Figure 3 Described in detail.

[0043] CPU 208 may be implemented using any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. CPU 108 may be a general purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM, or MIPS ISAs, or any other suitable ISA. Figure 2A single CPU is shown in FIG, but SOC component 204 may include multiple CPUs. In a multi-processor system, each CPU may collectively implement the same ISA, but this is not required.

[0044] Graphics processing unit (GPU) 220 is a graphics processing circuit for performing graphics data. For example, GPU 220 may render objects to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). GPU 220 may include one or more graphics processors that may execute graphics software to perform some or all graphics operations or hardware acceleration of certain graphics operations.

[0045] I / O interface 218 is hardware, software, firmware, or a combination thereof for interfacing with various input / output components in device 100. I / O components may include devices such as keyboards, buttons, audio devices, and sensors such as a global positioning system. I / O interface 218 processes data for sending data to such I / O components or processes data received from these I / O components.

[0046] The network interface 210 is a subcomponent that enables data to be exchanged between the device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data may be received from other devices via the network interface 210 and stored in the system memory 230 for subsequent processing (e.g., via a back-end interface to the image signal processor 206, such as described below in Figure 3 The network may include, but is not limited to, a local area network (LAN) (e.g., an Ethernet or a corporate network) and a wide area network (WAN). Image data received via network interface 210 may be processed by ISP 206 for image processing.

[0047] The sensor interface 212 is a circuit for interfacing with the motion sensor 234. The sensor interface 212 receives sensor information from the motion sensor 234 and processes the sensor information to determine the orientation or movement of the device 100.

[0048] The display controller 214 is a circuit for sending image data to be displayed on the display 216. The display controller 214 receives image data from the ISP 206, the CPU 208, the graphic processor 220, or the system memory 230, and processes the image data into a format suitable for display on the display 216.

[0049] Memory controller 222 is a circuit for communicating with system memory 230. Memory controller 222 can read data from system memory 230 for processing by ISP 206, CPU 208, GPU 220, or other subcomponents of SOC component 204. Memory controller 222 can also write data to system memory 230 received from various subcomponents of SOC component 204.

[0050] Video encoder 224 is hardware, software, firmware, or a combination thereof, for encoding video data into a format suitable for storage in persistent storage 128 or for passing the data to network interface 210 for transmission over a network to another device.

[0051] In some embodiments, one or more subcomponents of SOC component 204 or some functions of these subcomponents may be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, persistent storage 228, or another device that communicates with device 100 via network interface 210.

[0052] Image data or video data may flow through various data paths within SOC component 204. In one example, raw image data may be generated from image sensor 202 and processed by ISP 206, and then sent to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, it may be accessed by video encoder 224 for encoding, or by display 116 for display via bus 232.

[0053] In another example, image data is received from a source other than image sensor 202. For example, video data may be streamed, downloaded, or otherwise transferred to SOC component 204 via a wired or wireless network. Image data may be received via network interface 210 and written to system memory 230 via memory controller 222. The image data may then be obtained from system memory 230 by ISP 206 and processed through one or more image processing pipeline stages, as described below with reference to FIG. Figure 3 The image data may then be returned to system memory 230 or sent to video encoder 224 , display controller 214 (for display on display 216 ), or storage controller 226 for storage in persistent storage 228 .

[0054] Exemplary Image Signal Processing Pipeline

[0055] Figure 3 is a block diagram illustrating an image processing pipeline implemented using ISP 206 according to one embodiment. Figure 3In an embodiment of the present invention, ISP 206 is coupled to image sensor 202 to receive raw image data. ISP 206 implements an image processing pipeline, which may include a set of stages for processing image information from creation, capture, or reception to output. ISP 206 may include, among other components, a sensor interface 302, a central control 320, a front-end pipeline stage 330, a back-end pipeline stage 340, an image statistics module 304, a vision module 322, a back-end interface 342, and an output interface 316. ISP 206 may include Figure 3 Other components not shown or may be omitted Figure 3 One or more components shown in .

[0056] In one or more embodiments, different components of ISP 206 process image data at different rates. Figure 3 In an embodiment of the present invention, the front-end pipeline stages 330 (e.g., the raw processing stage 306 and the resampling processing stage 308) can process image data at the initial rate. Therefore, various different techniques, adjustments, modifications, or other processing operations are performed by these front-end pipeline stages 330 at the initial rate. For example, if the front-end pipeline stages 330 process 2 pixels per clock cycle, the raw processing stage 308 operations (e.g., black level compensation, highlight recovery, and defective pixel correction) can process 2 pixels of image data at a time. In contrast, one or more back-end pipeline stages 340 can process image data at a different rate that is less than the initial data rate. For example, in Figure 3 In some embodiments, the back-end pipeline stages 340 (e.g., the noise processing stage 310, the color processing stage 312, and the output rescaling 314) may be processed at a reduced rate (e.g., 1 pixel per clock cycle). Although the embodiments described herein include embodiments in which one or more back-end pipeline stages 340 process image data at a rate different from the initial data rate, in some embodiments, the back-end pipeline stages 340 may process image data at the initial data rate.

[0057] The sensor interface 302 receives raw image data from the image sensor 202 and processes the raw image data into image data that can be processed by other stages in the pipeline. The sensor interface 302 can perform various pre-processing operations, such as image cropping, merging, or scaling to reduce the image data size. In some embodiments, pixels are sent from the image sensor 202 to the sensor interface 302 in raster order (i.e., horizontally, row by row). Subsequent processing in the pipeline can also be performed in raster order, and the results can also be output in raster order. Although Figure 3Only a single image sensor 202 and a single sensor interface 302 are shown in FIG. 1 . When more than one image sensor is provided in the device 100 , a corresponding number of sensor interfaces may be provided in the ISP 206 to process raw image data from each image sensor.

[0058] The front-end pipeline stage 330 processes the image data in raw or full color gamut. The front-end pipeline stage 330 may include, but is not limited to, a raw processing stage 306 and a resampling processing stage 308. For example, the raw image data may be in a Bayer raw format. In a Bayer raw image format, pixel data with values ​​specific to a particular color (rather than all colors) is provided in each pixel. In image capture sensors, image data is typically provided in a Bayer pattern. The raw processing stage 308 may process the image data in a Bayer raw format.

[0059] The operations performed by the raw processing stage 308 include, but are not limited to, sensor linearization, black level compensation, fixed pattern noise reduction, defective pixel correction, raw noise filtering, lens shading correction, white balance gain, and highlight recovery. Sensor linearization refers to mapping nonlinear image data to a linear space for other processing. Black level compensation refers to independently providing digital gain, offset, and clipping for each color component of the image data (e.g., Gr, R, B, Gb). Fixed pattern noise reduction refers to removing offset fixed pattern noise and obtaining fixed pattern noise by subtracting a dark frame from the input image and multiplying different gains by the pixels. Defective pixel correction refers to detecting defective pixels and then replacing the defective pixel values. Raw noise filtering refers to reducing the noise of image data by averaging neighboring pixels of similar brightness. Highlight recovery refers to estimating the pixel values ​​of those pixels that are clipped (or close to clipping) from other channels. Lens shading correction refers to applying gain to each pixel to compensate for the intensity drop that is roughly proportional to the distance from the optical center of the lens. White balance gain refers to independently providing digital gain, offset, and clipping for white balance for all color components (e.g., Gr, R, B, Gb in Bayer format). Components of ISP 206 may convert raw image data to image data in the full color gamut, and thus raw processing stage 308 may process image data in the full color gamut in addition to or in lieu of raw image data.

[0060] The resampling processing stage 308 performs various operations to convert, resample or scale the image data received from the raw processing stage 306. The operations performed by the resampling processing stage 308 may include, but are not limited to, demosaicing operations, per-pixel color correction operations, gamma mapping operations, color space conversions, and downscaling or sub-band splitting. A demosaicing operation refers to converting or interpolating color-missing samples from raw image data (e.g., in a Bayer pattern) into output image data into a full color domain. A demosaicing operation may include low-pass directional filtering of the interpolated samples to obtain full-color pixels. A per-pixel color correction operation refers to a process that uses information about the relative noise standard deviation of each color channel to perform color correction on a per-pixel basis to correct the color without amplifying noise in the image data. Gamma mapping refers to converting image data from input image data values ​​to output data values ​​to perform special image effects, including black and white conversion, sepia tone conversion, negative conversion, or exposure conversion. For the purpose of gamma mapping, a lookup table (or other structure that indexes a pixel value to another value) for different color components or channels for each pixel may be used (e.g., separate lookup tables for Y, Cb, and Cr color components). Color space conversion refers to converting the color space of the input image data into a different format. In one embodiment, the resampling processing stage 308 converts the RBD format to a YCbCr format for further processing.

[0061] The central control 320 may control and coordinate the overall operation of the other components in the ISP 206. The central control 320 performs various operations including, but not limited to, monitoring various operating parameters (e.g., recording clock cycles, memory latency, quality of service, and status information), updating or managing control parameters of other components of the ISP 206, and interfacing with the sensor interface 302 to control the start and stop of other components of the ISP 206. For example, the central control 320 may update the programmable parameters of other components while the other components in the ISP 206 are in an idle state. After updating the programmable parameters, the central control 320 may place these components of the ISP 206 in a running state to perform one or more operations or tasks. The central control 320 may also instruct the other components of the ISP 206 to store image data before, during, or after the resampling processing stage 308 (e.g., by writing Figure 2 In this manner, full resolution image data in an original or full color gamut format may be stored in addition to or in lieu of processing image data output from the resampling processing stage 308 through the back end pipeline stage 340.

[0062] The image statistics module 304 performs various operations to collect statistical information associated with the image data. Operations to collect statistical information may include, but are not limited to, sensor linearization, mask pattern defective pixels, sub-sampling raw image data, detecting and replacing non-patterned defective pixels, black level compensation, lens shading correction, and inverted black level compensation. After performing one or more such operations, statistical information such as 3A statistics (auto white balance (AWB), auto exposure (AE), auto focus (AF)), histograms (e.g., 2D color or component), and any other image data information may be collected or tracked. In some embodiments, when a previous operation identifies clipped pixels, the values ​​of certain pixels or regions of pixel values ​​may be excluded from the collection of certain statistical data (e.g., AF statistics). Although Figure 3 Only a single statistics module 304 is shown in FIG. 2 , multiple image statistics modules may be included in ISP 206. In such embodiments, each statistics module may be programmed by central control 320 to collect different information for the same or different image data.

[0063] The visual module 322 performs various operations to facilitate computer vision operations at the CPU 208, such as face detection in image data. The visual module 322 may perform various operations, including preprocessing, global tone mapping and γ correction, visual noise filtering, resizing, key point detection, convolution, and generation of directional gradient histograms (HOG). If the input image data is not in YCrCb format, the preprocessing may include subsampling or merging operations and brightness calculations. Global mapping and γ correction may be performed on the preprocessed data on the brightness image. Visual noise filtering is performed to remove pixel defects and reduce the noise present in the image data, thereby improving the quality and performance of subsequent computer vision algorithms. Such visual noise filtering may include detection and fixation of point or defect pixels, and bilateral filtering to reduce noise by averaging adjacent pixels of similar brightness. Various visual algorithms use images of different sizes and scales. For example, resizing of the image is performed by merging or linear interpolation operations. Key points are locations within an image that are surrounded by image blocks that are very suitable for matching other images of the same scene or object. Such key points are useful in image alignment, calculating camera poses, and object tracking. Keypoint detection refers to the process of identifying such keypoints in an image. Convolution can be used in image / video processing and machine vision. For example, convolution can be performed to generate edge maps of an image or to smooth an image. HOG provides a description of image patches for tasks in image analysis and computer vision. For example, HOG can be generated by (i) computing horizontal and vertical gradients using a simple difference filter, (ii) computing gradient directions and magnitudes from horizontal and vertical gradients, and (iii) merging the gradient directions.

[0064] The back-end interface 342 receives image data from other image sources other than the image sensor 202 and forwards it to other components of the ISP 206 for processing. For example, the image data may be received via a network connection and stored in the system memory 230. The back-end interface 342 retrieves the image data stored in the system memory 230 and provides it to the back-end pipeline stage 340 for processing. One of the many operations performed by the back-end interface 342 is to convert the retrieved image data into a format that can be used by the back-end processing stage 340. For example, the back-end interface 342 may convert image data formatted in RGB, YCbCr 4:2:0, or YCbCr 4:2:2 to a YCbCr 4:4:4 color format.

[0065] The back-end pipeline stage 340 processes the image data according to a specific full-color format (e.g., YCbCr 4:4:4 or RGB). In some embodiments, components of the back-end pipeline stage 340 may convert the image data to a specific full-color format before further processing. The back-end pipeline stage 340 may include other stages such as the noise processing stage 310 and the color processing stage 312. The back-end pipeline stage 340 may include Figure 3 Other stages not shown.

[0066] The noise processing stage 310 performs various operations to reduce noise in the image data. The operations performed by the noise processing stage 310 include, but are not limited to, color space conversion, gamma / de-gamma mapping, temporal filtering, noise filtering, luminance sharpening, and chrominance noise reduction. Color space conversion can convert image data from one color space format to another color space format (e.g., RGB format converted to YCbCr format). The gamma / de-gamma operation converts image data from input image data values ​​to output data values ​​to perform special image effects. Temporal filtering uses previously filtered image frames to filter out noise to reduce noise. For example, the pixel values ​​of the previous image frame are combined with the pixel values ​​of the current image frame. Noise filtering can include, for example, spatial noise filtering. Luminance sharpening can sharpen the luminance values ​​of the pixel data, while chrominance suppression can attenuate the chrominance to gray (i.e., no color). In some embodiments, luminance sharpening and chrominance suppression can be performed simultaneously with spatial noise filtering. The aggressiveness of noise filtering can be determined differently for different regions of the image. Spatial noise filtering can be included as part of a time loop to implement temporal filtering. For example, a previous image frame may be processed by a temporal filter and a spatial noise filter before being stored as a reference frame for the next image frame to be processed. In other embodiments, spatial noise filtering may not be included as part of the temporal loop for temporal filtering (e.g., a spatial noise filter may be applied to an image frame after the image frame is stored as a reference image frame (and therefore not a spatially filtered reference frame)).

[0067] The color processing stage 312 can perform various operations associated with adjusting the color information in the image data. The operations performed in the color processing stage 312 include, but are not limited to, local tone mapping, gain / offset / clipping, color correction, three-dimensional color lookup, gamma conversion, and color space conversion. Local tone mapping refers to a spatially varying local tone curve to provide more control when rendering an image. For example, a two-dimensional grid of tone curves (which can be programmed by the central control 320) can be bilinearly interpolated, thereby producing a smoothly varying tone curve on the image. In some embodiments, local tone mapping can also apply a spatially varying and intensity-varying color correction matrix, which can be used, for example, to make the sky bluer while lowering the blue in the shadows in the image. Digital gain / offset / clipping can be provided for each color channel or component of the image data. Color correction can apply a color correction transformation matrix to the image data. 3D color lookup can perform advanced tone mapping, color space conversion, and other color transformations using a three-dimensional array of color component output values ​​(e.g., R, G, B). For example, gamma conversion can be performed by mapping input image data values ​​to output data values ​​to perform gamma correction, tone mapping, or histogram matching. Color space conversion may be implemented to convert image data from one color space to another (e.g., RGB to YCbCr). Other processing techniques may also be performed as part of the color processing stage 312 to perform other special image effects, including black and white conversion, sepia tone conversion, negative conversion, or light exposure conversion.

[0068] As the ISP 206 processes the image data, the output rescaling module 314 can resample, transform, and correct distortion on the fly. The output rescaling module 314 can calculate fractional input coordinates for each pixel and use the fractional coordinates to interpolate the output pixels via a polyphase resampling filter. The fractional input coordinates can be generated from a variety of possible transformations of the output coordinates, such as resizing or cropping the image (e.g., via simple horizontal and vertical scaling transformations), rotating and shearing the image (e.g., via inseparable matrix transformations), perspective warping (e.g., via an additional depth transform), and per-pixel perspective segmentation applied in strip-like segments to account for changes in the image sensor during image data capture (e.g., due to a rolling shutter), as well as geometric distortion correction (e.g., via calculating the radial distance from the optical center to index an interpolated radial gain table, and applying a radial perturbation to the coordinates to account for radial lens distortion).

[0069] When processing image data at the output rescaling module 314, the output rescaling module 314 may apply a transformation to the image data. The output rescaling module 314 may include a horizontal scaling component and a vertical scaling component. The vertical portion of the design may implement a series of image data row buffers to maintain the "support" required for the vertical filter. Since the ISP 206 can be a streaming device, there may only be image data rows in a limited length sliding window of rows available for the filter. Once a row is discarded to make room for a new incoming row, the row may be unavailable. The output rescaling module 314 may statistically monitor the calculated input Y coordinates on the previous row and use it to calculate a set of optimal rows to be kept in the vertical support window. For each subsequent row, the output rescaling module may automatically generate a guess about the center of the vertical support window. In some embodiments, the output rescaling module 314 may implement a piecewise perspective transformation table encoded as a digital differential analyzer (DDA) stepper to perform a per-pixel perspective transformation between the input image data and the output image data to correct artifacts and motion caused by sensor motion during the capture of the image frame. The output rescaling may provide the image data to various other components of the system 100 via the output interface 314, as described above with respect to Figure 1 and Figure 2 Discussed.

[0070] In various embodiments, the functions of components 302 to 342 can be similar to Figure 3 The order in which these functional units in the image processing pipeline are shown in FIG. 1 may be different from the order implied by the order in which these functional units are executed or may be different from the order in which the functional units are executed. Figure 3 The functional components shown in FIG. are implemented by different functional components. Figure 3 The various components described in the specification may be embodied in various combinations of hardware, firmware or software.

[0071] Exemplary Vision Module

[0072] As mentioned above Figure 3 As described above, the vision module 322 performs various operations to facilitate computer vision operations at the CPU 208. To this end, the vision module 322 may include, among other components, a histogram of oriented gradients (HOG) module 412, a multiplexer 420, and a convolution engine 414, such as Figure 4 As shown. The visual module 322 may include Figure 4 Other components not shown, such as a scaling module.

[0073] The HOG engine 400 processes the images to generate HOG data 426 for each image. An example of HOG data 426 is a histogram of directional gradients generated for the image based on the gradient directions identified within the image. The HOG data 426 can be used for various computer vision applications, such as image classification, scene detection, facial expression detection, human detection, object detection, scene classification, and text classification.

[0074] The multiplexer 420 receives HOG data 426 from the HOG engine 412 and pixel data 424 from components of the image processing processor 206 other than the HOG engine 412 (e.g., DRAM memory), and selects the HOG data 426 or the pixel data 424 as input data 422 to be forwarded to the convolution engine 414 according to various operating modes. In one mode, the multiplexer 420 may forward the HOG data 426 as input data 422 to the convolution engine 414. In another mode, the multiplexer 420 may forward the pixel data 424 as input data 422 to the convolution engine 414 for performing operations such as sharpening, blurring, and edge detection. Configuration signals for controlling the multiplexer 420 may be received from the central control 320. The pixel data 422 is a stream of interleaved pixel values ​​for multiple channels.

[0075] The convolution engine 414 is a configurable circuit that performs a convolution operation on the input data 422. To this end, the convolution engine 414 includes components for storing convolution kernel information, for performing calculations, and for accumulating multiplied values ​​to generate output 428, as described below in conjunction with Figure 5 Described in detail.

[0076] Figure 4 The structure of the visual module 322 shown is only exemplary and may be used for Figure 4 Various changes may be made to the structure of the convolution engine 414. For example, components such as the HOG engine 412 and the multiplexer 420 may be omitted. Alternatively, the multiplexer 420 may receive pixel data from more than two sources and select one source for input to the convolution engine 414 as stream input data 422.

[0077] In the following description, for ease of explanation, it is assumed that the input data 422 is a pixel value. However, it should be noted that the input data 422 may be other types of data suitable for convolution operations (eg, HOG data).

[0078] Example Convolution Engine Architecture

[0079] Figure 5is a block diagram illustrating a convolution engine 414 according to one embodiment. The convolution engine 414 is a circuit that performs operations on interleaved multi-channel image data to facilitate image / video processing and computer vision. The convolution engine 414 can perform various types of operations on the multi-channel image data, such as convolution operations, inter-channel processing operations, and per-channel processing operations. Exemplary convolution operations may include generating an edge map or a smoothed image. For example, an image convolved with a Gaussian kernel may produce a smoothed image with reduced noise and aliasing. As another example, when an image is convolved with a set of multi-directional convolution kernels, the convolution engine 414 generates image features for classification, such as Gabor features. In addition, in some embodiments, the convolution engine 414 facilitates template matching for deep machine learning classification tasks (such as person or object detection).

[0080] The convolution engine 414 performs various operations to facilitate convolutional neural network (CNN) tasks, such as spatial pooling and local response normalization. CNN is a deep learning architecture that can perform image classification, object detection, and other computer vision tasks.

[0081] The convolution engine 414 receives input data 422 , such as input data from bus 232 , and performs a convolution operation on the input data 422 based on stored convolution kernel information, performs inter-channel and per-channel processing on the results of the convolution operation, and generates output data 428 .

[0082] Convolution engine 414 may include, among other components, a first convolution circuit 502, a demultiplexer circuit 503, a second convolution circuit 504, and a channel merging circuit 506. Although convolution engine 414 is illustrated as including first convolution circuit 502 and second convolution circuit 504, in some embodiments, convolution engine 414 may include N convolution circuits. First convolution circuit 502 receives a stream of input data 422 and applies one or more convolution kernels to input data 422 to generate a stream of values ​​530. Second convolution circuit 504 also receives a stream of input data 422 (or alternatively, a stream of values ​​530 output from first convolution circuit 502) and applies one or more convolution kernels to input data 422 to generate a stream of values ​​532. The streams of input data received and processed by convolution circuits 502 or 504 each define one or more channels of input data in an interleaved manner.

[0083] The first convolution circuit 502 includes a multiplexer 508, a pre-processing circuit 510, a convolution core circuit 512, and a spatial merging circuit 514. The multiplexer 508 is coupled to the pre-processing circuit 510, the pre-processing circuit 510 is coupled to the convolution core circuit 512, and the convolution core circuit 512 is coupled to the spatial merging circuit 514.

[0084] The multiplexer 508 receives the input data 422 from the bus 232 and provides the input data 422 to the pre-processing circuit 510. In some embodiments, the multiplexer 508 selects between the input data 424 from the bus 232 and one or more other data sources (e.g., HOG data 426) and provides the selected data to the pre-processing circuit 510. In other embodiments, the multiplexer 508 is omitted from the first convolution circuit 502, and the pre-processing circuit 510 receives the input data 424 from the bus 232.

[0085] The preprocessing circuit 510 performs preprocessing operations on the interleaved input data 422, such as by applying gain, offset and clipping operations to the input data 422. These operations can be used to apply various types of processing before convolution, such as mean subtraction or contrast stretching. In some embodiments, the preprocessing circuit 510 identifies the value of each channel from the stream of input data 422, and processes each channel independently to apply different gain, offset or clipping operations to the input values ​​of different channels. For example, the input data 422 can be a Bayer raw format, including interleaved Gr, R, B and Gb channels. The preprocessing circuit 510 can apply different gain, offset or clipping operations to the pixel data of different channels. In some embodiments, the preprocessing circuit 510 operates in a bypass mode, which passes the input to the convolution core circuit 512 without applying preprocessing operations.

[0086] The convolution core circuit 512 receives preprocessed input data from the preprocessing circuit 510 and applies one or more convolution kernels to the input data. The convolution core circuit 512 may also perform post-processing on the convolution result. Post-processing may include operations that generate values ​​for deep machine learning, such as multi-channel normalized cross-correlation (NCC) or inter-channel local response normalization (LRN). Multi-channel or inter-channel operations combine values ​​from two or more channels for the convolution result. The sequence of values ​​in the stream generated by the convolution core circuit 512 defines multiple data channels in an interleaved manner. The result of the convolution core circuit 512 is provided to the spatial merging circuit 514. In some embodiments, the result of the convolution core circuit 512 is output from the convolution engine 414, as shown by the stream of values ​​436.

[0087] Spatial merging circuit 514 performs per-channel operations on the output of convolution core circuit 512, such as per-channel spatial merging and per-channel local response normalization (LRN), and outputs a stream of values ​​530. The per-channel operations process the values ​​associated with each channel separately. The per-channel LRN normalizes the local contrast in the response map. The per-channel operations can be applied after the convolution layer to facilitate deep machine learning. Compared to the convolution layer, the per-channel operations of the spatial merging circuit 514 have a lower computational cost because they are typically applied in a smaller local window and do not use the convolution kernel coefficients.

[0088] The second convolution circuit 504 includes a multiplexer 518, a preprocessing circuit 520, a convolution core circuit 522, and a spatial merging circuit 525. The above discussion about the first convolution circuit 502 can be applied to the second convolution circuit 504, as well as any other convolution circuit that can be included in the convolution engine 414. MUX 518 selects between the stream of values ​​530 output from the first convolution core circuit 502 and the input value 422 from the bus 232, and sends the selected input to the preprocessing circuit 520. The second convolution circuit 504 can apply similar operations to the stream of values ​​530 or the stream of input values ​​424, because both streams include channels of data in an interleaved manner. The operation and function of the preprocessing circuit 520, the convolution core circuit 522, and the spatial merging circuit 524 are substantially the same as the preprocessing circuit 510, the convolution core circuit 512, and the spatial merging circuit 514, and therefore, for the sake of brevity, a detailed description of these circuits is omitted herein.

[0089] Demultiplexer circuit 503 is a circuit that receives the stream of values ​​530 and sends the output to either channel merging circuit 506 or second convolution circuit 504. Demultiplexer 503 can select a route based on instructions from central control 320. Central control 320 sets the selection of demultiplexer 503 based on the operating mode, which is a cascade mode where convolution circuits 502 and 504 operate in series, and other modes where convolution circuits 502 and 504 operate in parallel. In some embodiments, demultiplexer circuit 503 is omitted from convolution engine 414.

[0090] An input of channel merging circuit 506 is coupled to the output of first convolution circuit 502 and another input is coupled to the output of second convolution circuit 504. Channel merging circuit 506 receives streams of values ​​530 and 532 from convolution circuits 502 and 504, respectively, and combines these values ​​into one or more output streams of output values ​​428, such as streams of values ​​428a and 428b. The streams of values ​​received from convolution circuits 502 and 504 may be processed in various ways depending on the operating mode selected for convolution engine 414, such as in conjunction with FIG. 6A to FIG. 6C discussed in more detail.

[0091] Channel merging circuit 506 includes channel merger 526 and multiplexer 528. Channel merger 526 receives the stream of values ​​530 from first convolution core circuit 502 and the stream of values ​​532 from second convolution core circuit 504, and interleaves values ​​530 and 532 to generate a stream of values ​​534. Stream of values ​​534 includes channels from the stream of values ​​530 and channels from the stream of values ​​532, as combined in an interleaved manner by channel merger 526.

[0092] Multiplexer 528 has an input coupled to the stream of values ​​534 from channel merger 526 and an input coupled to the stream of values ​​532 from second convolution core circuit 504. Multiplexer 528 selects between the stream of values ​​534 and the stream of values ​​532 to output as the stream of values ​​428b. Channel merger circuit 506 may also output the stream of values ​​530 from first convolution circuit 502 as the stream of values ​​428a.

[0093] Convolution engine processing mode

[0094] The convolution engine 414 operates in a variety of modes, including dual convolution mode, cascade mode, and parallel mode. The central control 320 sends configuration information to the convolution engine 414, which configures the convolution engine 414 to operate in a specified mode. The configuration information includes instructions for the components of the convolution engine 414. The configuration information can also specify the inputs and functions of the components, such as the convolution kernels used by each convolution circuit 502 and 504.

[0095] Fig. 6A A block diagram illustrating a dual convolution mode of a convolution engine 414 according to one embodiment. In dual convolution mode, convolution circuits 502 and 504 operate in parallel by applying different convolution kernels to the same input data 424, and a channel merging circuit 506 combines the results from convolution circuits 502 and 504 to generate an output stream of output values ​​428b. The attributes associated with each convolution kernel may include filter element values, kernel sizes of the convolution kernel (e.g., the height and width of the window defined by the kernel, in pixels), sparse values ​​of the convolution kernel, and step values ​​between convolutions. The stream of values ​​530 output from the first convolution circuit 502 and the stream of values ​​532 output from the second convolution circuit 504 are input to the channel merging circuit 506. The channel merging circuit 506 generates an output stream of output values ​​428b by interleaving the output values ​​530 and 532. The output stream of values ​​428b is output from the convolution engine 414 in dual convolution mode.

[0096] In some embodiments, the central control 320 sets the convolution engine 414 to dual convolution mode by controlling the selections made by multiplexers 508, 518, 528 and demultiplexer 503 to route data streams to data streams. Demultiplexer 503 (if used) routes the stream of values ​​530 to channel merge circuit 506, which is output as an output stream of output values ​​428a. Multiplexer 518 selects the stream of values ​​530 output from the first convolution circuit 502 as input to the second convolution circuit 504. Multiplexer 528 selects the stream of values ​​534 output from channel merge 526 as output stream of output values ​​428b. Multiplexer 508 (if used) selects input data 424 from bus 232 as input to the first convolution circuit 502.

[0097] Dual convolution mode is a configuration of the convolution engine 414 that processes the same input stream in parallel on two convolution core circuits 502 and 504. The convolution circuits 502 and 504 may apply different convolution kernels to the input data. To facilitate the computation of many output channels, the convolution engine 414 distributes processing tasks between the convolution circuits 502 and 504. For example, the first convolution circuit 502 may process the first half of the output channels, while the second convolution circuit 504 may process the second half of the output channels. The channel merging circuit 506 combines the streams of multiple channels from the convolution circuits 502 and 504 into a single stream having channels from the two streams interleaved with each other.

[0098] In some embodiments, each convolution circuit 502 and 504 has two execution clusters, each of which generates one pixel per clock (ppc). Therefore, each convolution circuit 502 and 504 generates two ppcs. After merging the channels in dual convolution mode, channel merging circuit 506 combines the results of convolution circuits 502 and 504 to generate a four ppc output.

[0099] Figure 6B 4 is a block diagram illustrating a cascade mode of a convolution engine 414 according to one embodiment. In cascade mode, convolution circuits 502 and 504 operate in series. First convolution circuit 502 applies one or more convolution kernels to input data 422 from bus 232 to generate a stream of values ​​530. Second convolution circuit 504 receives the stream of values ​​530 and applies one or more second convolution kernels to values ​​530 to generate a stream of values ​​532. Convolution circuits 502 and 504 may use different convolution kernels. Channel merging circuit 506 passes the stream of values ​​532 from convolution circuit 504, thereby generating an output stream of output values ​​428b.

[0100] Central control 320 places convolution engine 414 in cascade mode by controlling the selections made by multiplexers 508, 518, 528 and demultiplexer 503. Multiplexer 508 (if used) selects input data 424 from bus 232 as input to first convolution circuit 502. Demultiplexer 503 (if used) routes a stream of values ​​530 to multiplexer 518. Multiplexer 518 selects a stream of values ​​530 output from convolution engine 414 as input to second convolution circuit 504. Multiplexer 528 selects a stream of values ​​532 output from second convolution circuit 504 as an output stream of output values ​​428b.

[0101] In cascade mode, convolution circuits 502 and 504 perform two convolution operations in series without memory transfers between these operations. The first convolution circuit 502 in the cascade generates one ppc, such as by using only one of the two execution clusters. If the first convolution circuit 502 generates two ppc output streams, the subsequent second convolution circuit 504 will require two processes and four ppcs. Therefore, a single execution cluster is used in the first convolution circuit 502 to generate one ppc stream that is input to the second convolution circuit 504. The second convolution circuit 504 generates two ppc output streams from the one ppc stream of the first convolution circuit 502.

[0102] Figure 6C A block diagram illustrating a parallel mode of a convolution engine 414 according to one embodiment. In parallel mode, convolution circuits 502 and 504 operate in parallel as two single units to generate two separate interleaved output streams. For example, an image can be divided into two vertical strips, and each convolution circuit 502 and 504 processes one strip. Convolution circuits 502 and 504 can process the same input data or different input data. When the input data is the same, convolution circuits 502 and 504 can apply different convolution kernels to the input data. For another example, convolution circuits 502 and 504 apply different kernels to different input data.

[0103] The first convolution circuit 502 applies one or more convolution kernels to the input data 422 from the bus 232 to generate a stream of values ​​530. The second convolution circuit 504 applies one or more second convolution kernels to the input data 422 from the bus 232 to generate a stream of values ​​532. The channel merging circuit 506 generates an output stream of output values ​​428a by passing the flow of values ​​530, and generates an output stream of output values ​​428b by passing the flow of values ​​532. The separate output streams 428a and 428b can each define multiple data channels in an interleaved manner. In parallel mode, the outputs from the convolution circuits 502 and 504 are maintained in separate streams of interleaved channels, rather than being combined into a single stream of interleaved channels.

[0104] The central control 320 sets the convolution engine 414 to parallel mode by controlling the selection at multiplexers 508, 518, 528 and demultiplexer 503. Multiplexer 508 (if used) selects input data 422 from bus 232 as input to the first convolution circuit 502. Demultiplexer 503 (if used) routes the stream of values ​​530 from the output of the first convolution circuit 502 to the channel merge circuit 506 for output as an output stream of output values ​​428a. Multiplexer 518 selects input data 422 from bus 232 as input to the second convolution circuit 504. Multiplexer 528 selects the stream of values ​​532 output from the second convolution circuit 504 for the output stream of output values ​​428b of the channel merge circuit 506. The channel merge circuit 506 also passes the stream of values ​​530 to the output stream of output values ​​428a.

[0105] In parallel mode, each convolution circuit 502 and 504 can generate two ppcs using two execution clusters. Channel merging circuit 506 outputs a first stream from the first convolution circuit 502 at two ppcs and outputs a second stream from the second convolution circuit 504 at two ppcs.

[0106] Figure 7 4 is a flow chart illustrating a method of operating the convolution engine 414 in multiple modes according to one embodiment. The central control 320 sends 702 configuration information to the convolution engine 414. The configuration information may include parameters of components of the convolution engine for placing the convolution engine in a particular operating mode, such as a dual convolution mode, a cascade mode, or a serial mode.

[0107] The configuration information may also include information defining the flow of values ​​input to each convolution core circuit 502 and 504. For example, the configuration information may define an image size and / or channel count so that components of the convolution engine 414 may identify pixels from each channel of the serialized flow.

[0108] The configuration information may also include information defining one or more convolution kernels used by each convolution kernel circuit 502 and 504, such as filter element values, kernel size, sparsity value, and stride value. The configuration information defining the convolution kernel specifies the convolution operation performed by each convolution kernel circuit 502 and 504.

[0109] After receiving the configuration information, the configuration of the convolution engine 414 is updated 704 according to the configuration information to perform the operations described in the configuration information. Updating the configuration may include routing flows within the convolution engine according to the selected operating mode. Routing control may be set using multiplexers 508, 518, and 528 of the convolution engine 414, as described above in conjunction with FIG. 6A to FIG. 6CAs discussed. Updating the configuration may include providing convolution kernels to convolution circuits 502 and 504. Convolution circuits 502 and 504 may also be configured to use one or two execution clusters, depending on the mode of operation as described above.

[0110] In some embodiments, the configuration instructions may also define one or more deep learning operations to be performed on the convolution results at each convolution circuit 502 and 504 before channel merging. Exemplary operations may include normalized cross-correlation calculations, response rectification, spatial merging, and local response normalization. In some embodiments, inter-channel operations may be performed by post-processing circuits 704 of convolution core circuits 502 and 504, and per-channel operations are performed by spatial merging circuits 514 and 524.

[0111] The first convolution circuit 502 of the convolution engine 414 generates 706 a first stream of values ​​by applying one or more first convolution kernels to the first input data. The second convolution circuit 504 of the convolution engine 414 generates 708 a second stream of values ​​by applying one or more second convolution kernels to the second input data. Generating the first input data and the second input data may include performing a convolution, and may also include applying one or more deep learning operations using post-processing circuitry of the convolution core circuit 512 / 522 or the spatial merging circuit 514 / 524.

[0112] In the dual convolution mode, the first input data and the second input data used by the convolution circuits 502 and 504 may be the same, and the first convolution kernel and the second convolution kernel may be different. In the cascade mode, the second input data used by the second convolution circuit 504 is the output of the first convolution circuit 502, and the first convolution kernel and the second convolution kernel may be different. In the parallel mode, the first input data and the second input data may be the same, and the first convolution kernel and the second convolution kernel may be different.

[0113] The channel merging circuit generates 710 one or more output streams based on the first stream of values ​​from the first convolution circuit 502 and the second stream of values ​​from the second convolution circuit 504. In the dual convolution mode, the channel merging circuit 710 generates the output stream by combining the interleaved first stream of interleaved channel values ​​from the first convolution circuit 502 with the second stream of interleaved channel values ​​from the second convolution circuit 504 in an interleaved manner. In the cascade mode, the channel merging circuit 710 generates an output stream including the second stream of interleaved channel values ​​from the second convolution circuit 504, wherein the second stream of interleaved channel values ​​is derived by applying one or more second convolution kernels to the first stream of interleaved channel values ​​at the second convolution circuit 504. In the series mode, the channel merging circuit 710 generates a first output stream including the first stream of interleaved channel values ​​from the first convolution circuit 502 and a separate second output stream including the second stream of interleaved channel values ​​from the second convolution circuit 504.

[0114] like Figure 7 The process shown is merely illustrative, and various changes may be made to the process. For example, as specified by the configuration information and operating mode of the convolution engine 414, the first stream of values ​​generated 706 and the second stream of values ​​generated 708 may be performed in parallel or in series.

[0115] Convolution core circuit

[0116] Figure 8 8 is a block diagram illustrating a convolution core circuit 800 according to one embodiment. The convolution core circuit 800 is an example of the convolution core circuit 512 of the first convolution circuit 502 or the convolution circuit 522 of the second convolution circuit 504, as shown in FIG. Figure 5 As shown. Convolution core circuit 800 includes convolution core 802 and post-processing circuit 804. Convolution core 802 receives input data 836 and performs a convolution operation by applying one or more convolution kernels h to input data 836. Input data 836 may be input data 422 from bus 323, output of another convolution circuit, or input data from some other source, and may be pre-processed by pre-processing circuit 510 as described above. Post-processing circuit 804 performs post-processing on the output of convolution core 802.

[0117] The convolution core circuit 802 includes a convolution front end 806, a core memory 808, an execution cluster 810, an execution cluster 812, and a convolution back end 814. The convolution front end 806 is coupled to the execution clusters 810 and 812. The convolution front end 806 receives input data 836 and prepares the input data 836 for processing by the execution clusters 810 and 812. The convolution front end 806 distributes processing tasks involving the input data and the convolution kernel between the execution clusters 810 and 812.

[0118] Each execution cluster 810 and 812 is coupled to the convolution front end and the core memory 808. Each execution cluster 810 and 812 may include multiple multiplication and accumulation (MAC) units. When multiple output channels are used, output channels with even indices may be processed by one execution cluster, while output channels with even indices may be processed by another execution cluster. Each execution cluster 810 and 812 may generate one ppc, so the convolution core 802 as a whole may generate two ppcs. Execution cluster 810 generates a stream of even data values ​​842 including even indexed output channels and a stream of odd data values ​​844 including odd indexed output channels.

[0119] The core memory 808 stores one or more convolution kernels h provided to the execution clusters 810 and 812. In some embodiments, the central control 320 provides one or more convolution kernels h to the core memory 808 to control the convolution operation. Each execution cluster 810 and 812 applies the convolution kernel from the core memory 808 to the input data 836 prepared by the convolution front end 806. The execution clusters 810 and 812 can execute in parallel to generate output values, for example, at two ppcs. In some embodiments, only a single execution cluster 810 or 812 is enabled to generate output values, for example, at one ppc.

[0120] In one example, execution clusters 810 and 812 apply a series of convolution kernels to different portions of the input data to generate a stream of even data values ​​842 including even-indexed output channels and a stream of odd data values ​​844 including odd-indexed output channels. The even data values ​​842 and the odd data values ​​844 represent multi-channel data that are independently processed in a post-processing pipeline using inter-channel operations such as local response normalization and normalized cross-correlation.

[0121] In some embodiments, convolution front end 806 generates kernel statistics 840 for the convolution kernels, which are stored in kernel memory 808 and processed by execution clusters 810 and 812. The kernel statistics can be derived from the properties of the convolution kernels. Kernel statistics 840 can include ∑H and ∑H 2 , where H is the kernel data of the convolution kernel. The convolution core 802 sends the kernel statistics 840 to the post-processing circuit 804.

[0122] The convolution backend 814 is coupled to the outputs of the execution clusters 810 and 812. The convolution backend 814 performs further processing on the output values ​​from each execution cluster. Such operations may include, but are not limited to, multi-cycle accumulators for large bit data.

[0123] In some embodiments, the convolution backend 814 or some other component of the convolution core 802 generates local statistics based on the input data 836. The local statistics may include ∑I, ∑I 2and ∑I*H, where I is the input data 836 and H is the convolution kernel applied to the input data 836. In some embodiments, the local statistics are transmitted to the post-processing circuit 804 via a stream of even data values ​​842 and a stream of odd data values ​​844. For example, the local statistics may be an auxiliary channel of streams 842 and 844, such as the last active channel of a multi-channel stream. In other embodiments, the local statistics may be transmitted in the stream with the kernel statistics 840, or in a separate stream.

[0124] Thus, convolution kernel 802 generates a stream of even data values ​​842, a stream of odd data values ​​844, kernel statistics 840, and local statistics. These values ​​are provided to post-processing circuitry 804 for additional processing. Fig.10 Exemplary circuitry for convolution core 802 is discussed in more detail.

[0125] The post-processing circuit 804 includes a processing pipeline for each execution cluster 810 and 812 to process the corresponding output streams 842 and 844. To process the stream 842 from the execution cluster 810, the post-processing circuit 804 includes a multi-channel normalized cross-correlation (NCC) unit 816, a response rectifier unit 818, an inter-channel local response normalization (LRN) unit 820, and an output generation unit 822. To process the stream 844 from the execution cluster 812, the post-processing circuit 804 includes a multi-channel NCC unit 824, a response rectifier unit 826, an inter-channel LRN unit 828, and an output generation unit 830. The post-processing circuit 804 may also include a peak finder 843, a demultiplexer 832, and a core merger 846.

[0126] The multi-channel NCC unit 816 calculates the NCC score and the normalized kernel statistics for the stream of even data values ​​842. The multi-channel NCC unit 816 is coupled to the convolution core 802 to receive the stream of even data values ​​842, the local statistics, and the kernel statistics 840. The multi-channel NCC unit 816 determines the NCC score for each convolution kernel based on the even data values ​​842, the local statistics, and the kernel statistics 840.

[0127] The multi-channel NCC unit 816 can calculate the NCC score for each convolution kernel. The NCC score is normalized by the input variance defined by the local statistics and the variance of the kernel defined by the kernel statistics. The NCC score can be used to find the best correspondence between two frames.

[0128] For each convolution kernel, the NCC score can be defined by Formula 1:

[0129]

[0130] Where I is the input data, H is the kernel data, M I and M His the average value of I and H, σ I and σ H is the standard deviation of I and H, and N is the size of the convolution kernel. Additional scaling and offset factors can be applied to avoid division by zero and reduce quantization errors.

[0131] The multi-channel NCC unit 816 may also calculate normalized kernel statistics. For example, the multi-channel NCC unit 816 calculates kernel statistics as defined in Formula 2:

[0132] (NΣH 2 -(∑H) 2 ) (2)

[0133] Where N is the size of the convolution kernel and H is the kernel data. Formula 2 forms part of the denominator of Formula 1, so the kernel statistics can be calculated in the process of calculating the NCC score.

[0134] The normalized kernel statistics are scaled versions of the kernel statistics processed using a scaling factor. The scaling factor can be defined by Equation 3:

[0135]

[0136] Where N is the size of the convolution kernel. The scaling factor normalizes the kernel statistics to be independent of the kernel size. The multi-channel NCC unit 816 sends the normalized kernel statistics 852 and / or NCC scores to the peak finder 834.

[0137] The response rectifier unit 818 is coupled to the multi-channel NCC unit 816. The response rectifier unit 818 receives the stream of data values ​​842 and performs a non-linear transformation on the data values ​​842. The non-linear transformation facilitates deep machine learning for describing high-level features. The stream of data values ​​842 input to the response rectifier unit can be transmitted from the multi-channel NCC unit 816. In some embodiments, the multi-channel NCC unit 816 is omitted from the post-processing circuit 804, and the response rectifier unit 818 receives the stream of data values ​​842 from the execution cluster.

[0138] Fig. 9 904 is a graph of a nonlinear transformation applied by a response rectifier unit 818 according to some embodiments. The response rectifier unit 818 receives as input a stream of values ​​842 and applies an offset parameter 912 to the values ​​842. The offset parameter 912 may be selected to model a bias applied after a convolutional layer in a deep learning architecture. After applying the offset, the response rectifier unit 818 applies scaling to negative input values ​​based on a configurable scaling factor 904. The response rectifier unit 818 outputs a stream of rectified data values. In some embodiments, the response rectifier unit 818 clips negative values ​​to zero. In other embodiments, the response rectifier unit 818 converts negative values ​​to positive values.

[0139] See again Figure 8 , the response rectifier unit 818 may apply different offset and scaling parameters to different channels. The parameters of the response rectifier unit 818 may be specified by the central control 320. In some embodiments, the central control 320 may disable the response rectifier unit 818. Here, the response rectifier unit 818 may be used as a bypass for the flow of values ​​in the post-processing pipeline. In some embodiments, the response rectifier unit 818 is omitted from the post-processing circuit 804.

[0140] The inter-channel LRN unit 820 is coupled to the response rectifier unit 818 and performs inter-channel LRN on the output of the response rectifier unit 818. Specifically, the inter-channel LRN unit 820 receives the stream of data values ​​842 and local statistics, and performs local response normalization to generate a normalized convolution output stream of data values. The inter-channel LRN unit 820 facilitates processing for deep learning architectures. The inter-channel LRN unit 1200 may perform a fixed-point approximation of the operation defined by Formula 4:

[0141]

[0142] where x i is the pixel index value, α is the normalized intensity, i' is approximately x i The index of the pixel within the local window of N is the number of pixels in the window. The support for local windows is inter-channel and is therefore represented as a rectangular region in a flattened format. The inter-channel LRN unit 820 performs inter-channel LRN in a post-processing stage to utilize the serial flow of interleaved channels, while per-channel normalization is handled separately such as by the spatial merging circuit 514.

[0143] The output generation unit 822 is coupled to the inter-channel LRN unit 820. The output generation unit 822 applies scaling, offset, and shift to the output of the inter-channel LRN unit 820.

[0144] The post-processing pipeline for the stream of odd values ​​844 may operate substantially the same as the processing pipeline for the stream of even values ​​842, so a detailed description of these circuits is omitted herein for the sake of brevity.

[0145] Core merger 846 combines even stream 842 and odd stream 844 having even channels and odd channels (e.g., after post-processing) into a stream of data values ​​848 including even channels and odd channels in an interleaved manner. Core merger 847 is coupled to output generation unit 822 and output generation unit 830.

[0146] The post-processing circuit 804 may also include a demultiplexer 832. The demultiplexer 832 is coupled to the output generation unit 830 and selectively provides the stream of values ​​from the output generation unit 830 to the core merger 846 (for combination into the output stream 848) or provides it as an output stream 850. The stream of values ​​848 combines the pixel values ​​from both the execution clusters 810 and 812, and thus the core merger 846 generates the output, for example, with two ppcs. The stream of values ​​850 is generated using only the values ​​from the execution cluster 812, and thus may be generated, for example, with one ppc. As described above in conjunction with FIG. 6A to FIG. 6C As discussed, the convolution core circuit 800 may be configured to generate one PPC or two PPCs in different operating modes of the convolution engine 414.

[0147] The peak finder 834 is coupled to the multi-channel NCC unit 816 and the output generation unit 822 of the first post-processing pipeline, and is coupled to the multi-channel NCC unit 824 and the output generation unit 830 of the second post-processing pipeline. In some embodiments, the normalized kernel statistics can be used as a confidence measure of the reliability of the template matching result. The peak finder 834 receives the normalized kernel statistics 852 and the convolution result, and determines the position that provides the best match for the template based on the NCC score. The peak finder 843 determines the position based on a predetermined criterion. For example, the peak finder 843 can find the minimum pixel position or the maximum pixel position for the selected channel. When a list of high-dimensional feature vectors is given as input data, the peak finder can find the vector closest to the origin based on the distance metric evaluated by the convolution kernel.

[0148] In some embodiments, the peak finder 834 monitors the stream of data from the output generation units 822 and 830. For the selected channel, the peak finder 834 accesses each value of the channel in the stream to track the position with the minimum or maximum value. The selected output channel may contain an NCC score or any other convolution result. If the channel contains an NCC score (e.g., the multi-channel NCC unit 816 is enabled for the selected channel), the peak finder 834 outputs a normalized kernel statistic with a peak position and a peak NCC score. If NCC is not enabled, the peak finder 834 outputs the peak position and the peak value.

[0149] In some embodiments, the central control 320 sends configuration information to the convolution core 802 and the post-processing circuit 804 of the convolution core circuit 800. The configuration instructions may include post-processing instructions for each pipeline of each post-processing circuit 804 and define the post-processing to be applied to the convolution results from the convolution core 802.

[0150] The post-processing instruction defines whether to enable or disable the multi-channel NCC unit, the response rectifier unit, the inter-channel LRN unit, or the peak finder. In some embodiments, the post-processing circuit 804 operates in a plurality of modes specified by the post-processing instruction. In NCC mode, the multi-channel NCC unit is enabled and the inter-channel LRN unit is disabled. In LRN mode, the multi-channel NCC unit is disabled and the inter-channel LRN unit is enabled. In hybrid LRN / NCC mode, the multi-channel NCC unit and the inter-channel LRN unit are enabled. In pass-through mode, the multi-channel NCC unit and the inter-channel LRN unit are disabled. The disabled components in the post-processing pipeline can pass their input data stream to the next component in the post-processing pipeline without processing the stream.

[0151] Fig.10 802, according to one embodiment. As described above, convolution core 802 includes circuits such as convolution front end 806, execution clusters 810 and 812, and convolution back end 814.

[0152] The convolution front end 806 may include an input buffer 1002, a data path router 1006, a sequencer 1018, and a kernel statistics unit 1024. The input buffer 1002 stores input data 836 as it is streamed to the convolution front end 806. The input data 836 may be a stream of values, with data for multiple input channels in an interleaved manner. The input data 836 may be pixel data, HOG data, an output of a previous cycle of the convolution circuit 800, an output of another convolution circuit 800, or other data received from other components of the device 100.

[0153] The data path router 1006 is a circuit that reads a set of data 1004 in a predetermined position of the input buffer 1002 in a scan order, and sends the read data 1008 to the execution cluster 810 or 812 for calculating the convolution value. The data path router 1006 can send different parts of the input data 836 to the execution clusters 810 and 812 to utilize the convolution kernel for parallel processing. The scan sequence described herein refers to the operation of processing a subset of the input data. The data path router 1006 can perform the reading and sending of data of multiple scan sequences within the processing cycle of the convolution engine 414 to fill the execution clusters 810 and 812 with pixel values. In one embodiment, the data path router 1006 selectively reads the pixel value of the center pixel and the pixel value of the pixel subset adjacent to the center pixel, while skipping other adjacent pixels according to the sparse value. In addition, the center pixel to be processed in the scan sequence can be separated by a plurality of pixels defined by the step value. In subsequent scans, a new set of center pixels separated by the same or different number of pixels can be processed.

[0154] The kernel memory 808 is a circuit for storing kernel information. The kernel information includes the value of the filter element in the convolution kernel, the sparse value, the stride value, the kernel size, etc. The kernel information 1022 is sent to the execution cluster 810 to fill the registers in the multiplier circuits FE0 to FEN of the execution cluster 810. The kernel information 1022 is also sent to the execution cluster 812 to fill the registers in the multiplier circuits FE0 to FEN of the execution cluster 812. The kernel memory 808 can store multiple convolution kernels for performing convolution with different channels of pixel data and / or performing convolution with the same channel of pixel data.

[0155] Execution clusters 810 and 812 are programmable circuits that perform computational operations. To this end, execution clusters 810 and 812 may include multiplier circuits FE0 to FEN, compressor 1010, and multi-cycle accumulator 1014. Each of multiplier circuits FE0 to FEN may store a pixel value in read data 1008 and a corresponding filter element value in core memory 808. The pixel value and the corresponding filter element value are multiplied in the multiplier circuit to generate a multiplied value 1009. In some embodiments, compressor 1010 receives multiplied value 1009 and accumulates a subset of multiplied value 1009 to generate compressed value 1012. In other embodiments, instead of accumulating a subset of multiplied value 1009, compressor 1010 may select (i) a minimum value, (ii) a maximum value, or (iii) a median value from each subset of multiplied value 1009. The multi-cycle accumulator 1014 receives the compressed values ​​1012 and performs accumulation (or selects the minimum, maximum, or median) on the compressed values ​​1012 generated over multiple processing cycles of the convolution kernel 802.

[0156] Returning to the convolution front end 806, the sequencer 1018 controls the operation of the other components of the convolution core 802 to perform multiple operation cycles. The sequencer 1018 can effectively distribute processing tasks between the execution clusters 810 and 812. As described above, the execution clusters 810 and 812 apply a series of convolution kernels to different portions of the input data to generate a stream of even data values ​​842 including even indexed output channels and a stream of odd data values ​​844 including odd indexed output channels. For example, the core memory 808 provides a series of filter elements of the convolution kernel for each set of pixel data stored in the multiplier circuits FE0 to FEN. Each convolution kernel generates a different output channel of even data values ​​842 and odd data values ​​844.

[0157] In another exemplary operation of the sequencer 1018, the size of the input data and / or the number or size of the convolution kernels may be too large to perform all calculations in a single processing cycle of the execution cluster. The sequencer 1018 divides the calculation operations between the even and odd output channels, assigning the processing tasks of the even channels to the execution cluster 810 and the processing tasks of the odd channels to the execution cluster 812.

[0158] In some embodiments, the size of the input data and / or the number or size of the convolution kernels may be too large to perform all calculations in a single processing cycle of the convolution core 802 using both execution cores. In such cases, the sequencer 1018 divides the computational operations into multiple batches and performs computations based on a subset of the input data or a subset of the convolution kernels in a single cycle. The computation results in each cycle are processed by the multi-cycle accumulator 1014 to generate output values ​​1013 across multiple cycles. To configure other components to perform multi-cycle operations, the sequencer 1018 sends multi-cycle control signals 1019 to the other components.

[0159] The convolution backend 814 includes an output buffer 1024, a big data processor 1028, and an output buffer 1030, as well as a big data processor 1032. The output buffer 1024 is a circuit that stores the output value 1013 at its designated location. In one embodiment, a series of output values ​​of multiple output channels are interleaved in the output buffer 1024. In the operation of feeding back the output value 1015 of the execution cluster 810 as the input data 836 at the convolution front end 806 again, the data in the output buffer 1024 can be copied to the input buffer 1002 for the next cycle of the convolution operation. The output buffer 1024 processes the output value 1013 of the execution cluster 810, and the output buffer 1030 processes the output value 1013 of the execution cluster 812.

[0160] The big data processor 1032 is a circuit that further processes the output values ​​stored in the output buffer 1024. For example, the convolution kernel 802 can process input data and convolution kernels with different bit sizes (such as 8-bit or 16-bit precision). When the input data or convolution kernel has 16-bit precision, each output pixel will use twice the number of clock cycles. When both the input data and the convolution kernel have 16-bit precision, more than four times the clock cycle will be used. The convolution backend 814 can merge the results of the 8-bit pixel data convolution from multiple clock cycles into data with 16-bit precision. The big data processor 1032 can perform similar processing on the output values ​​from the execution cluster 812 stored in the output buffer 1024. The stream of even data values ​​842 is output from the big data processor 1028, and the stream of odd data values ​​844 is output from the big data processor 1032. In some embodiments, the big data processors 1028 and 1032 are omitted from the convolution backend 814. Streams of even data values ​​842 and odd data values ​​844 are output from output buffers 1024 and 1030, respectively. Smaller data sizes can support faster processing for machine reasoning tasks or other tasks that can use lower precision data. In contrast, larger data sizes can be used for machine training or higher precision tasks.

[0161] The components in the convolution core 802 (as well as other components of the convolution engine 414) may be configured during configuration by receiving configuration information from the central control 320. The configurable parameters and modes indicated in the configuration information may include, but are not limited to, sparsity values, stride values, mappings between pixel data values ​​and filter elements, the type of operation to be performed at the compressor 1010 (e.g., accumulation, minimum, maximum, or median), the number of channels of input data or output values, and selection of post-processing operations to be performed at the post-processing circuit 804.

[0162] Fig.10 The structure of the convolution kernel 802 in FIG. 1 is merely illustrative. For example, the multi-cycle accumulator 1014 may be omitted, thereby performing only a single-cycle operation on the convolution engine.

[0163] Fig.11A is a conceptual diagram illustrating the input and output of a multi-plane format convolution core circuit 800 according to one embodiment. The convolution core circuit 800 convolves multi-channel input data 1102 and generates multi-channel output data 1110. The number of input channels and output channels may be different. Fig.11A The multi-plane format shown in represents each input channel and output channel as a separate image plane. Multi-channel input data 1102 has pixel values ​​for three input channels 1104, 1106, and 1108. Each input channel 1104, 1106, and 1108 can be processed using one or more kernels. For example, Fig.11AThe four convolution kernels shown, kernel 0 to kernel 3, are applied to channel 1106 to produce multi-channel output data 1110 including four output channels 1112, 1114, 1116, and 1118. If the same four convolution kernels 0 to 3 (e.g., using sparse kernels) are applied to each input channel 1104, 1106, and 1108 for each channel, the multi-channel output will include four channels for each processed input channel, so there are twelve output channels in total. A different convolution kernel can be used to generate each different output channel. The size, sparsity value, and stride value of the convolution kernel can be flexible to allow different types of convolution to be used for different applications.

[0164] Fig. 11B 1 is a conceptual diagram showing the input and output of the planarized format convolution core circuit 800 according to one embodiment. The multi-channel input data 1102 and the multi-channel output data 1110 are respectively defined by a stream of n interleaved multiple channels, where the corresponding pixel value of each channel (represented by Fig. 11B ) are adjacent to each other in the stream, followed by the corresponding pixel values ​​for each channel of the next pixel, and so on, in a raster style as shown in the flattened format. The flattened format includes images of multiple interleaved channels from a single image plane represented as interleaved channels.

[0165] The multi-channel input data 1102 is defined by a stream, where related pixel values ​​from different channels are adjacent to each other in a planarized format. For example, the first channel pixel 1124, the second channel pixel 1126, and the third channel pixel 1128 represent the first (0,0) pixel of the input image defined by the multi-channel input data 1102. The pixel value of the next pixel (0,1) of the multi-channel input data 1102 follows pixels 1124, 1126, and 1128. The next pixel (0,1) includes the first channel pixel 1130, the second channel pixel 132, and the third channel pixel 1134. The subsequent pixels in the first row (0) can follow the (0,1) pixel accordingly. The pixel value of the subsequent row (1) can follow the pixel value of the first row. For example, the first pixel (1,0) in the second row includes the first channel pixel 1136, followed by the second channel pixel 1138, and then the third channel pixel 1140.

[0166] For example, the input channels of the multi-channel input data 1102 include RGB color channels. For another example, the multi-channel input data 1102 may include YCbCr color channels. For another example, the multi-channel input data 1102 may include an output channel of a convolution result derived using a convolution kernel.

[0167] Multi-channel output data 1110 is derived from multi-channel input data 1102 by applying a convolution kernel, such as convolution kernel 1150. Multi-channel output data 1100 includes a stream of related pixel values ​​from different output channels that are adjacent to each other in the stream, as shown in the flattened format. For example, output channel pixels 1142, 1144, 1146, and 1148 correspond to the (0,0) pixel of output data 1110. Fig.11A As shown, output channel pixels 1142, 1144, 1146, and 1148 belong to output channels 1112, 1114, 1116, and 1118, respectively. Thus, the serial stream may define interleaved channels of output data 1110 in a raster fashion.

[0168] When the convolution engine 414 operates in cascade mode, the convolution core circuit 800 uses the output of another convolution core circuit 800 as input, as shown in the above combination Figure 6B As discussed, the multi-channel input data 1102 and the multi-channel output data 1110 of the convolution core circuit 800 have a common interleaved format to facilitate multiple operation modes, including a mode of using the output data of a convolution core circuit 800 as input data of another convolution core circuit 800 .

[0169] Per-channel spatial pooling and normalization

[0170] Fig.12 1 is a block diagram illustrating a spatial merging circuit 1200 according to one embodiment. The spatial merging circuit 1200 performs per-channel spatial merging or normalization operations on a stream having multiple interleaved channels and also generates an output stream of multiple interleaved channels. Figure 5 As described, the convolution circuits 502 and 504 include a spatial merging circuit 514 and a spatial merging circuit 524, respectively, to process the output streams of the corresponding convolution core circuits 512 and 522. The spatial merging circuit 1200 is an implementation of the spatial merging circuit 512 of the first convolution circuit 502 or the spatial merging circuit 524 of the second convolution circuit 504. According to the instruction from the central control 320, the spatial merging circuit 1200 performs per-channel spatial merging and / or per-channel local response normalization on some or all input interleaved channels.

[0171] The spatial merging circuit 1200 includes an input buffer 1202, a per-pixel calculation block 1204, a column compressor 1206, a column accumulation buffer 1208, a row compressor 1210, a delay 1222, and a spatial merging and normalization (SPN) processor 1212. The SPN processor 1212 includes a square root unit 1214, a local response normalization unit (LRN) 1216, a multiplexer 1218, and an SPN post-processor 1220.

[0172] The input buffer 1202 receives a stream of values ​​1232 from the convolution core circuit 512 and stores the input data. The input data includes a stream of data values ​​defining multiple channels in an interleaved manner, and these data values ​​are stored in the input buffer 1202 when they are received in the stream. The input buffer 1202 stores multiple pixel values ​​of the same channel to facilitate per-channel processing. In order to generate spatially merged pixels, the size of the input buffer 1202 is set to store input pixel values ​​that are at least sufficient to fit the local window. In a flattened format in which pixel values ​​from multiple interleaved channels are represented as a single image plane, the local window has the sparsity of selecting pixel values ​​only for a single channel. The size (e.g., height or width) of the local window, which defines the number of pixel values ​​of the same channel to be spatially merged, can be configurable, such as by instructions from the central control 320. The horizontal stride of the local window defines the pixel spacing between the center pixels of the local window, and can also be configurable, such as by instructions from the central control 320. Because input buffer 1202 receives a stream of interleaved channels, where pixel values ​​of a channel are separated by pixel values ​​of one or more other channels, input buffer 1202 stores multiple pixel values ​​for each of the multiple channels.

[0173] The local window may include multiple pixel values ​​of a single channel to be spatially merged into a spatially merged pixel value. For each spatially merged pixel, the spatial merging circuit 1200 performs column merging to merge the pixel values ​​from a column of the local window, and then performs row merging to merge the column-merged values ​​of the local window. It should be noted that "row" and "column" refer to vertical pixel lines of a flattened image, and are not necessarily in a specific horizontal or vertical direction.

[0174] For each spatially binned pixel, per-pixel computation 1204 retrieves data values ​​for the channels of the local window from input buffer 1202 and performs an operation on the data values. The operation may include applying an offset to the data value, squaring the data value, or determining the absolute value of the data value.

[0175] The column compressor 1206 combines multiple data values ​​from the per-pixel calculations 1204 associated with a column of the local window into a single spatially merged value representing the column. The column compressor 1206 can combine the multiple data values ​​in a variety of ways, as can be specified by the central control 320. For example, the column compressor 1206 can select the minimum value, the maximum value, or can combine the values ​​into a sum.

[0176] The column accumulation buffer 1208 receives a plurality of spatially merged column values ​​from the column compressor 1204 and stores the spatially merged column pixel values. For example, the column accumulation buffer 1208 stores at least the spatially merged column values ​​for each column of the local window.

[0177] Row compressor 1210 combines the spatially merged column values ​​for each column of the local window. Similar to column compressor 1206, row compressor 1210 can combine multiple data values ​​in various ways, as specified by central control 320. For example, row compressor 1210 can select the minimum value, the maximum value, or can combine these values ​​into a sum. The output of row compressor 1210 represents the spatially merged value derived from each pixel of the local window.

[0178] The SPN processor 1202 processes the spatially merged values ​​received from the row compressor 1210. For example, the SPN processor 1202 may determine the square root of the spatially merged values. The SPN processor 1202 may alternatively or additionally use the spatially merged values ​​to perform local response normalization on the input stream 1222.

[0179] The SPN processor 1202 includes a square root unit 1214, an LRN unit 1216, a multiplexer 1218, and an SPN post-processor 1220. The square root unit 1214 calculates the square root of the spatially combined values ​​from the row compressor 1210.

[0180] The LRN unit 1216 performs local response normalization by applying the spatially combined values ​​from the row compressor to the input values ​​stored in the delay 1222 to generate a per-channel normalization value. The delay 1222 facilitates the local response normalization by synchronizing the spatially combined values ​​with the corresponding input values ​​from the input buffer 1202. The delay 1222 is coupled to the input buffer 1202 and the LRN unit 1216. The delay 1222 may include a first-in-first-out (FIFO) memory buffer.

[0181] Multiplexer 1218 selects an output from the spatially merged values ​​of row compressor 1210, selects the square root of the spatially merged values ​​from square root unit 1214, or selects a normalized value from LRN unit 1216. SPN post-processor 1220 receives the selected output of multiplexer 1218 and performs scaling, offset, and / or shifting operations. The output of SPN post-processor 1220 is a stream of pixel values ​​defining multiple channels in an interleaved manner, wherein the pixel values ​​are processed by per-channel spatial merging and / or per-channel normalization.

[0182] In some embodiments, the central control 320 operates the spatial consolidation circuit 1200 in different modes by configuring the operational combinations of components.

[0183] As above combined Figure 5 As described above, the output stream 530 of the spatial merging circuit 514 of the first convolution circuit 502 can be used as an input to the second convolution circuit 504, or can be provided to the channel merging circuit 506 to be interleaved with the output of the second convolution circuit 504.

[0184] Fig.13A and Fig. 13B 1 is a conceptual diagram illustrating the input and output of a multi-plane format spatial merging circuit 1200 according to one embodiment. The spatial merging circuit 1300 performs per-channel spatial merging and / or per-channel LRN on a multi-channel input image and generates a multi-channel output. The number of input channels and output channels is preserved, wherein the pixel image size of each image is reduced via spatial merging.

[0185] Fig.13A and Fig. 13B The multi-plane format of represents each input channel and output channel as a separate image plane. Multi-channel input data 1302 has pixel values ​​from multiple channels, such as channels 1304, 1306, and 1308. In this example, the pixel values ​​of a local window 1310 having a width and height of three pixels are spatially merged to generate a spatially merged value 1312 for output channel 1304. The spatial merging circuit 1200 generates multi-channel output data 1314 for channels 1304, 1306, and 1308 on a separate basis using the local window 1310 for each channel.

[0186] In such Fig.13A After calculating the first spatial merge value of the plurality of channels (eg, value 1312), as shown in Fig. 13B As shown, the local window 1310 is shifted to calculate the next spatial merged value (e.g., value 1322) of these channels. In this example, the local window 1310 is shifted by two pixels in the column dimension according to a raster pattern. This causes the center pixel of the local window 1310 to be shifted by two pixels in the column dimension. The amount of center pixel shift of the local window 1310 calculated per spatial merged pixel can be configurable. The local window can be shifted in a raster pattern according to predefined row ("stride X") and column ("stride Y") parameters of each spatial merged pixel until all spatial merged pixels are calculated. Using stride X and stride Y parameters greater than 1 will result in subsampling to reduce data size and computational cost. When these factors are equal to 1, output pixels are not skipped. The pixel values ​​of the shifted local window 1310 are spatially merged to generate the spatial merged value 1322 of the output channel 1316.

[0187] Fig. 13C and Fig.13D is a conceptual diagram illustrating the input and output of a planarized format spatial merge circuit 1300 according to one embodiment. Fig. 13C Corresponds to Fig.13A The multi-plane format shown in Fig.13D Corresponds to Fig. 13BIn the planarized format, each input channel is represented as a column of pixels placed at a horizontal interval of Cin, where Cin represents the number of input channels. Therefore, when per-channel operations are applied to the local window, the kernel support becomes sparse in the planarized format, as shown in the local window 1310.

[0188] In multi-planar format, the row ("stride X") and column ("stride Y") shift values ​​are defined in pixels in the channel's spatial coordinates. In planar format, the actual row shift amount is determined by multiplying the row shift value stride X by the number of input channels Cin.

[0189] Fig.14 14 is a flow chart illustrating a method 1400 of operating the spatial merging circuit 1200 according to one embodiment. The central control 320 sends 1402 configuration information to the spatial merging circuit 1200. The configuration instructions may be sent in conjunction with other configuration instructions for the convolution engine 414, as discussed at 702 of the method 700.

[0190] The configuration instructions may include instructions that define an operating mode of the spatial merging circuit 1200. Different operating modes may define different types of spatial merging or per-channel LRN. In a maximum merging mode, the column compressor 1206 and the row compressor 1210 select the maximum value, and the multiplexer 1218 selects the output of the row compressor 1210. Here, the post-accumulation processing of the SPN processor 1212 is bypassed so that the output of the spatial merging circuit 1200 does not have local response normalization or square root applied. In an average merging mode, the column compressor 1206 and the row compressor 1210 generate a sum, and the multiplexer 1218 selects the output of the row compressor 1210 to bypass the post-accumulation processing.

[0191] In the L1 merge mode, the per-pixel calculation 1204 determines an absolute value, the column compressor 1206 and the row compressor 1210 calculate the sum of the absolute values, and the multiplexer 1218 selects the output of the row compressor 1210 to bypass the post-accumulation process. In the L2 merge mode, the per-pixel calculation 1204 determines a square value, the column compressor 1206 and the row compressor 1210 calculate the sum of the square values, the square root unit 1214 determines the square root of the sum of the square values, and the multiplexer 1218 selects the output of the square root unit 1214.

[0192] In per-channel LRN mode, per-pixel calculation 1204 determines a square value, column compressor 1206 and row compressor 1210 calculate the sum of the square values, LRN unit 1216 normalizes the value using the square root of the sum of the square values, and multiplexer 1218 selects the output of LRN unit 1216.

[0193] After receiving the configuration information, the configuration of the spatial merging circuit 1200 is updated 1404 according to the configuration information to perform the operations described in the configuration information. Updating the configuration may include setting the operation of the per-pixel calculation 1204, the column compressor 1206 and the row compressor 1210, the square root unit 1214, and the multiplexer 1218 according to the operation mode defined by the configuration information.

[0194] The convolution core circuit 512 (or 522) generates 1406 streams of values ​​of multiple channels in an interleaved manner by performing convolution operations on input data. For example, the convolution core circuit 512 uses multiple convolution kernels to perform convolution operations on input data to generate a stream of values ​​including multiple channels according to configuration instructions. The convolution circuit 512 may further perform one or more post-processing operations on the convolution results specified by the configuration instructions. In some embodiments, the post-processing operations include inter-channel operations such as multi-channel NCC and inter-channel LRN. These operations combine values ​​from different channels and are different from the per-channel operations of the spatial merging circuit 1200. If the convolution core circuit 512 includes multiple execution clusters, the output streams of the multiple execution clusters may be combined to generate a stream of values ​​of multiple channels in an interleaved manner output by the convolution core circuit 512.

[0195] The spatial merging circuit 1200 generates 1408 spatial merged values ​​by merging subsets of values ​​from each channel with each other. For example, if the stream from the convolution core circuit 512 includes a first interleaved channel and a second interleaved channel, the spatial merging circuit 1200 generates a first spatial merged value by merging a subset of values ​​of the first channel (e.g., as defined by a local window), and generates a second spatial merged value by merging a subset of values ​​of the second channel. The input buffer 1202 ensures that a subset of values ​​from a single channel of the stream 1224 is stored to facilitate spatial merging. Based on the selected operations of the per-pixel calculation 1204, the column compressor 1206, the row compressor 1210, and the SPN processor 1212, subsets of values ​​from each channel can be merged in various ways. The spatial merged value may include values ​​derived from different types of spatial merging, such as a maximum merging mode, an average merging mode, an L1 merging mode, or an L2 merging mode. For another example, the spatial merged value may include a value derived from normalization such as a per-channel LRN mode.

[0196] The spatial merging circuit 1200 interleaves 1410 the spatially merged values ​​from the multiple channels in the output stream 1226. The spatial merging circuit 1200 thus maintains the multi-channel interleaved format received as the input stream 1224 at the output stream 1226 while performing per-channel deep machine learning operations on the input stream 1224.

[0197] The spatial merging circuit 1200 may receive a 2ppc input stream from the convolution core circuit 512 (or 504) and generate a 2ppc output stream. If the convolution core circuit 512 provides a 1ppc stream, the spatial merging circuit 1200 ignores invalid values ​​and processes only valid values. If the total width of the output frame is an odd number, a zero may be added at the end of each row to make the width uniform.

[0198] like Fig.14 The process shown is merely illustrative, and various changes may be made to the process. For example, in bypass mode, the spatial merging circuit 1200 may regroup the input stream to ensure a 2ppc output stream containing valid values. Pixel processing components such as per-pixel calculation 1204 and column compressor 1206 and row compressor 1210 may be bypassed in bypass mode.

[0199] Interleaved channel merging

[0200] When the application requires high throughput or when a large deep learning model is used, the two convolution circuits 502 and 504 can be operated in parallel in a dual convolution mode, as described above in conjunction with Fig. 6A As discussed. The two convolution circuits 502 and 504 apply different convolution kernels to the same input stream. For example, the first convolution circuit 502 generates the first half of the output channel with one or more convolution kernels, while the second convolution circuit 504 generates the second half with one or more different convolution kernels. The channel merging circuit 506 receives the streams from the convolution circuits 502 and 504 and combines the streams in an interleaved manner into a single output stream including the first half of the output channel and the second half of the output channel. In order to perform interleaving, the channel merging circuit has a channel merger 526.

[0201] Fig.15 is a block diagram illustrating a channel merger 1500 according to one embodiment. Channel merger 1500 is an embodiment of channel merger 526 of channel merger circuit 506. When operating in dual convolution mode, the output of channel merger 1500 is selected as the output of convolution engine 414.

[0202] Channel merger 1500 includes an input buffer 1502, a multiplexer 1504, and a channel selector 1506. Input buffer 1502 is coupled to convolution circuit 502 for receiving a stream of values ​​530 and convolution circuit 504 for receiving a stream of values ​​532. The streams of values ​​530 and 532 may each include multiple interleaved channels. Input buffer 1502 stores values ​​530 and 532 to facilitate synchronization of values ​​for interleaving.

[0203] Multiplexer 1504 is coupled to the input buffer and receives the stream of values ​​530 and 532 from input buffer 1502. Channel selector 1506 provides a selection signal to multiplexer 1504 to control the selection of values ​​from the input stream for insertion into the output stream of output values ​​534. The multiplexer interleaves the streams of values ​​530 and 532, such as by alternatively selecting one or more values ​​from each input stream to generate the output stream of output values ​​534. The number of sequential values ​​selected from a particular input stream may be defined by the number of channels per pixel in the stream. The sequence of output values ​​534 defines the channels of the streams of values ​​530 and 532 in an interleaved manner.

[0204] The channel merger 1500 supports two 2ppc input streams in sync without slowing down either input stream. The throughput of the merged output is 4ppc. If the two input streams are not synchronized, an input buffer 1502 can be used to store one or more input sources to provide a delay so that the channel merger 1500 receives synchronized input from the two input streams.

[0205] Fig.16 1 is a conceptual diagram showing the input and output of a planarized format channel merger 1500 according to one embodiment. In some embodiments, the channel merger 1500 combines two input frames of the same size, as shown in multi-channel input data 1602 and multi-channel input data 1604. In addition, the input streams 530 and 532 have the same number of input channels Cin. In this example, Cin is 5, so each pixel P0, P1, P2, etc. has a value of five channels for each stream. The channel merger 1500 generates multi-channel output data 1606 by interleaving multi-channel input data 1602 and multi-channel input data 1604, so that the pixel value of each channel of the P0 pixel of the first stream is followed by the pixel value of each channel of the P0 pixel of the second stream. In a raster manner of the planarized format, the pixel value of each channel of the P1 pixel of the first stream follows the pixel value of the P0 pixel of the second stream. For the P1 pixel, the pixel value of each channel of the P1 pixel of the first stream is followed by the pixel value of each channel of the P1 pixel of the second stream.

[0206] The channel merger 1500 generates an output stream of output values ​​534 having twice the number of channels as the number of input channels from each input stream 530 (including multi-channel input data 1602) and 532 (including multi-channel input data 1604). For example, each pixel P0, P1, etc. of the multi-channel output data has a 10-channel output Cout.

[0207] In some embodiments, when the height and width of the images in the input streams 530 and 532 do not match, or when the number of channels in the input streams 530 and 532 do not match, the channel merger 1500 is disabled in the channel merger circuit 506. The convolution engine 414 can operate in different modes that bypass the channel merger 1500, such as Figure 6B Cascade mode shown or Figure 6C The parallel mode shown here instead operates in dual convolution mode.

[0208] In some embodiments, channel merger 1500 is an embodiment of core merger 846. Core merger 847 receives two 1ppc input streams (after post-processing in separate pipelines) from each execution cluster 810 and 812, and combines these 1ppc input streams into a 2ppc output stream of the convolution core circuit 800. In contrast, channel merger 526 receives a 2ppc input stream and generates a 4ppc output stream. Therefore, channel merger 526 has a higher throughput than core merger 847. Core merger 847 may include: a multiplexer that selects data values ​​from even stream 842 and odd stream 844 to generate an output stream; and a channel selector that controls the selection of values ​​performed by the multiplexer. In some embodiments, core merger 846 may include one or more input buffers to facilitate interleaved synchronization by storing one or more of even stream 842 and odd stream 844. Due to the lower throughput, the size of the memory and processing components of the core merger 846 may be smaller than the size of the memory and processing components of the channel merger 1500.

Claims

1. A convolution engine, comprising: a first convolution circuit configured to generate a first stream of first values ​​by applying one or more first convolution kernels to first input data, the first values ​​in the first stream defining a plurality of first data lanes in an interleaved manner; as well as A second convolution circuit, coupled to the first convolution circuit, wherein the second convolution circuit is configured as follows: receiving the first value in the first stream as second input data from the first convolution circuit; as well as generating a second stream of second values ​​by applying one or more second convolution kernels to the second input data, the second values ​​in the second stream defining a plurality of second data channels in an interleaved manner, The second convolution circuit comprises: a first execution cluster configured to generate a first output value; a second execution cluster configured to generate a second output value; and A first post-processing circuit is coupled to the first execution cluster and the second execution cluster, the first post-processing circuit being configured to generate the second stream of second values ​​by combining the first output value and the second output value.

2. The convolution engine of claim 1 , wherein a first convolution kernel of the one or more first convolution kernels comprises a first filter element, and a second convolution kernel of the one or more second convolution kernels comprises a second filter element different from the first filter element.

3. The convolution engine of claim 1 , wherein the first convolution circuit comprises: a third execution cluster and a fourth execution cluster, the third execution cluster configured to generate a third output value when the fourth execution cluster is deactivated; and A second post-processing circuit is coupled to the third execution cluster and the fourth execution cluster, the second post-processing circuit being configured to generate the first stream of first values ​​from the third output value.

4. The convolution engine of claim 3, wherein: The third execution cluster is configured to generate one of the third output values ​​for each clock cycle; The first execution cluster is configured to generate one of the first output values ​​for each clock cycle; and The second execution cluster is configured to operate in parallel with the first execution cluster and to generate one of the second output values ​​per clock cycle.

5. The convolution engine of claim 1 , wherein the first post-processing circuit comprises a multi-channel normalized cross-correlation (NCC) unit configured to calculate a multi-channel NCC score for a plurality of second convolution kernels and the first output value from the first execution cluster.

6. The convolution engine of claim 1, wherein the first post-processing circuit further comprises a peak finder configured to determine a maximum value or a minimum value associated with the first output value from the first execution cluster.

7. The convolution engine of claim 1, wherein the first post-processing circuit further comprises a response rectifier unit configured to perform a non-linear transformation on the first output value from the first execution cluster.

8. The convolution engine of claim 1, wherein the first post-processing circuit comprises an inter-channel local response normalization (LRN) unit configured to perform inter-channel local response normalization on the first output value from the first execution cluster.

9. The convolution engine of claim 1, wherein the first input data comprises an input stream of input values ​​defining a plurality of channels of an image in an interleaved manner.

10. The convolution engine of claim 1, wherein the first convolution circuit is configured to provide the first value of the first stream to the second convolution circuit without storing the first value in a memory external to the first convolution circuit and the second convolution circuit.

11. A method for processing data, comprising: generating, by a first convolution circuit of a convolution engine, a first stream of first values ​​by applying one or more first convolution kernels to first input data, the first values ​​in the first stream defining a plurality of first data channels in an interleaved manner; receiving, by a second convolution circuit of the convolution engine coupled to the first convolution circuit, the first value of the first stream from the first convolution circuit as second input data; generating, by the second convolution circuit, a second stream of second values ​​by applying one or more second convolution kernels to the second input data, the second values ​​in the second stream defining a plurality of second data channels in an interleaved manner, generating, by a first execution cluster of the second convolution circuit, a first output value; generating, by a second execution cluster of the second convolution circuit, a second output value; and A second stream of second values ​​is generated by first post-processing circuitry coupled to the first execution cluster and the second execution cluster by combining the first output value and the second output value.

12. The method of claim 11, wherein a first convolution kernel of the one or more first convolution kernels comprises a first filter element, and a second convolution kernel of the one or more second convolution kernels comprises a second filter element different from the first filter element.

13. The method according to claim 11, further comprising: generating, by a third execution cluster of the first convolution circuit, a third output value when a fourth execution cluster of the first convolution circuit is disabled; and The first stream of first values ​​is generated from the third output value by second post-processing circuitry coupled to the third execution cluster and the fourth execution cluster.

14. The method according to claim 13, wherein: The third execution cluster generates one of the third output values ​​in each clock cycle; The first execution cluster generates one of the first output values ​​in each clock cycle; and The second execution cluster operates in parallel with the first execution cluster and generates one of the second output values ​​per clock cycle. 15 . The method of claim 11 , further comprising calculating, by a multi-channel normalized cross-correlation (NCC) unit of the first post-processing circuit, a multi-channel NCC score for a plurality of second convolution kernels and the first output value from the first execution cluster. 16 . The method of claim 11 , further comprising determining, by a peak finder of the first post-processing circuit, a maximum or minimum value associated with the first output value from the first execution cluster. 17 . The method of claim 11 , further comprising performing, by a response rectifier unit of the first post-processing circuit, a non-linear transformation on the first output value from the first execution cluster. 18 . The method of claim 11 , further comprising performing, by a channel-to-channel local response normalization (LRN) unit of the first post-processing circuit, channel-to-channel local response normalization (LRN) on the first output values ​​from the first execution cluster.

19. The method of claim 11, wherein the first input data comprises an input stream of input values ​​defining a plurality of channels of an image in an interleaved manner.

20. The method of claim 11, further comprising providing, by the first convolution circuit, the first value of the first stream to the second convolution circuit without storing the first value in a memory external to the first convolution circuit and the second convolution circuit.

21. A convolution engine comprising: a first convolution circuit configured to generate a first stream of first values ​​by applying one or more first convolution kernels to first input data, the first values ​​in the first stream defining a plurality of first data lanes in an interleaved manner; and a second convolution circuit configured to generate a second stream of second values ​​by applying one or more second convolution kernels to second input data, the second values ​​in the second stream defining a plurality of second data lanes in an interleaved manner, the first convolution circuit and the second convolution circuit operating in a parallel manner to generate the first stream of first values ​​and the second stream of second values; and a channel merging circuit, the channel merging circuit being coupled to the first convolution circuit to receive the first stream and being coupled to the second convolution circuit to receive the second stream, the channel merging circuit being configured to: generating, in a first mode of the convolution engine, a first output stream including the first values ​​and a second output stream including the second values, the first input data received by the first convolution circuit being the same as the second input data received by the second convolution circuit; as well as In a second mode of the convolution engine, a third output stream is generated including the first value interleaved with the second value, the third output stream defining the plurality of first data channels and the plurality of second data channels in an interleaved manner, the first input data received by the first convolution circuit being the same as the second input data received by the second convolution circuit.

22. A convolution engine according to claim 21, wherein the first convolution circuit applies a first convolution kernel to the first input data, and the second convolution circuit applies a second convolution kernel to the second input data, the first convolution kernel having a first filter element and the second convolution kernel having a second filter element different from the first filter element.

23. The convolution engine of claim 21 , wherein the first convolution circuit comprises: A convolution core, the convolution core comprising: a first execution cluster configured to generate a stream of even data values, the stream of even data values ​​defining an even lane of the plurality of first data lanes; and a second execution cluster configured to generate a stream of odd data values, the stream of odd data values ​​defining an odd lane of the plurality of first data lanes; and A post-processing circuit is coupled to the first execution cluster to receive the stream of even data values ​​and to the second execution cluster to receive the stream of odd data values, the post-processing circuit being configured to generate the first stream of first values ​​from the stream of even data values ​​and the stream of odd data values.

24. The convolution engine of claim 23, wherein the post-processing circuit comprises a multi-channel normalized cross-correlation (NCC) unit configured to calculate a multi-channel NCC score for the plurality of first convolution kernels and the stream of even data values.

25. The convolution engine of claim 24, wherein the post-processing circuit further comprises a peak finder configured to determine a best fit position of a template based on NCC scores associated with even channels.

26. The convolution engine of claim 23, wherein the post-processing circuit comprises a response rectifier unit configured to perform a non-linear transformation on the stream of even data values ​​from the first execution cluster.

27. The convolution engine of claim 23, wherein the post-processing circuit comprises an inter-channel local response normalization (LRN) unit configured to perform inter-channel local response normalization on the even data values ​​of the even channels defining the plurality of first data channels.

28. The convolution engine of claim 23, wherein: A post-processing circuit, the post-processing circuit comprising: a multi-channel normalized cross-correlation NCC unit coupled to the first execution cluster and configured to calculate NCC scores for the plurality of first convolution kernels and the stream of even data values; a response rectifier unit coupled to the multi-channel normalized cross-correlation (NCC) unit and configured to perform a non-linear transformation on a stream of even-numbered data values; and an inter-channel local response normalization LRN unit coupled to the response rectifier unit and configured to perform inter-channel local response normalization on an output of the response rectifier unit; and The convolution engine further includes a control circuit configured to selectively activate or deactivate the multi-channel normalized cross-correlation (NCC) unit, the response rectifier unit, and the inter-channel local response normalization (LRN) unit.

29. The convolution engine of claim 21, further comprising a control circuit configured to switch the convolution engine between a plurality of modes including the first mode and the second mode.

30. The convolution engine of claim 28, wherein: The second convolution circuit includes a first multiplexer; The channel merging circuit includes a second multiplexer; and The control circuit is further configured to: In the first mode of the convolution engine: causing the first multiplexer to select the first input data as the second input data of the second convolution circuit; and causing the second multiplexer to select the second stream of the second value as the second output stream of the channel merging circuit; as well as In the second mode of the convolution engine: causing the first multiplexer to select the first input data as the second input data of the second convolution circuit; as well as The second multiplexer is caused to select the third output stream as the output of the channel merging circuit.

31. A method for processing data, comprising: generating a first stream of first values ​​by applying one or more first convolution kernels to first input data, the first values ​​in the first stream defining a plurality of first data channels in an interleaved manner; generating a second stream of second values ​​by applying one or more second convolution kernels to second input data, the second values ​​in the second stream defining a plurality of second data channels in an interleaved manner, the first stream of first values ​​and the second stream of second values ​​being generated in parallel; generating, in a first mode of the convolution engine, a first output stream including the first values ​​and a second output stream including the second values, the first input data received by the first convolution circuit being the same as the second input data received by the second convolution circuit; as well as In a second mode of the convolution engine, a third output stream is generated including the first value interleaved with the second value, the third output stream defining the plurality of first data channels and the plurality of second data channels in an interleaved manner, the first input data received by the first convolution circuit being the same as the second input data received by the second convolution circuit.

32. A method according to claim 31, wherein the first convolution circuit applies a first convolution kernel to the first input data, and the second convolution circuit applies a second convolution kernel to the second input data, the first convolution kernel having a first filter element and the second convolution kernel having a second filter element different from the first filter element.

33. The method of claim 31 , further comprising, by the convolution engine: generating a stream of even data values, the stream of even data values ​​defining an even lane of the plurality of first data lanes; and generating a stream of odd data values, the stream of odd data values ​​defining odd channels of the plurality of first data channels; and The first stream of first values ​​is generated from the stream of even data values ​​and the stream of odd data values.

34. The method of claim 33, further comprising, by the convolution engine: Calculating a first multi-channel normalized cross-correlation (NCC) score for a plurality of first convolution kernels and the stream of even data values; and A second multi-channel NCC score is calculated for the plurality of first convolution kernels and the stream of odd data values.

35. The method of claim 34, further comprising determining, by the convolution engine, a best-fit position of a template based on NCC scores associated with even channels.

36. The method of claim 33, further comprising performing, by the convolution engine, a non-linear transformation on the stream of even data values.

37. The method of claim 33, further comprising performing, by the convolution engine, inter-channel local response normalization on the even data values ​​defining the even channels of the plurality of first data channels.

38. An electronic device comprising: A circuit arrangement is configured to: generating a first stream of first values ​​by applying one or more first convolution kernels to first input data, the first values ​​in the first stream defining a plurality of first data channels in an interleaved manner; generating a second stream of second values ​​by applying one or more second convolution kernels to second input data, the second values ​​in the second stream defining a plurality of second data channels in an interleaved manner, the first stream of first values ​​and the second stream of second values ​​being generated in parallel; generating, in a first mode of the circuit arrangement, a first output stream comprising the first value and a second output stream comprising the second value; as well as In a second mode of the circuit arrangement, a third output stream is generated comprising the first values ​​interleaved with the second values, the third output stream defining the plurality of first data lanes and the plurality of second data lanes in an interleaved manner.

39. The electronic device of claim 38, wherein: The first input data is the same as the second input data; and The one or more first convolution kernels are different from the one or more second convolution kernels.

40. The electronic device of claim 38, wherein the circuit arrangement comprises: a first execution cluster configured to generate a stream of even data values, the stream of even data values ​​defining an even lane of the plurality of first data lanes; a second execution cluster configured to generate a stream of odd data values, the stream of odd data values ​​defining an odd lane of the plurality of first data lanes; and A post-processing circuit is coupled to the first execution cluster to receive the stream of even data values ​​and to the second execution cluster to receive the stream of odd data values, the post-processing circuit being configured to generate the first stream of first values ​​from the stream of even data values ​​and the stream of odd data values.

Images