Application Processor including Neural Processing Unit and the Operating Method thereof

Abstract

An application processor according to an exemplary embodiment of the present disclosure includes a neural network processing device that converts an input image provided from an image sensor into a first image of a first pattern using a neural network and generates a second image to compensate for the conversion, and an image signal processing device having a plurality of pipelines that each receive the first image of the first pattern and the second image and perform image signal processing.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to an application processor, and more specifically, to an application processor having a structure for optimizing image signal processing and a method of operating the same. Background Technology

[0002] An image sensor can convert optical signals incident through an optical lens into electrical image signals. Recently, image sensors have been modified in structure depending on the product in which they are installed, generating images with various patterns. An application processor may include a processing unit that converts images generated by the image sensor into RGB format images. As the patterns of images generated by the image sensor become more diverse, there is a demand for an application processor architecture that optimizes the use of hardware resources when converting to RGB format images. The problem to be solved

[0003] The problem that the technical concept of the present disclosure aims to solve is to provide an application processor having a structure for optimizing image signal processing, and a method of operating the same. means of solving the problem

[0004] To achieve the above objectives, an application processor according to the technical concept of the present disclosure comprises: a neural network processing device that converts an input image provided from an image sensor into a first image of a first pattern using a neural network and generates a second image to compensate for the conversion; and an image signal processing device having a plurality of pipelines that each receive the first image of the first pattern and the second image and perform image signal processing.

[0005] An application processor according to the technical concept of the present disclosure includes a neural network processing device that generates a first image representing detail information for each frame from an input image provided by an image sensor using a neural network, and an image signal processing device that converts each frame image constituting the input image into a second image of a Bayer pattern and performs image signal processing for the second image based on the first image.

[0006] A method of operation of an application processor according to the technical concept of the present disclosure comprises the steps of receiving an input image, converting the input image into a first image of a Bayer pattern using a neural network, generating a second image to compensate for the conversion using a neural network, and performing image signal processing for the first image based on the second image. Effects of the invention

[0007] An application processor according to the technical concept of the present disclosure may include a neural network processing unit that converts an input image into a first image of a first pattern and generates a second image to compensate for the conversion. As the neural network processing unit is configured as a network having low complexity, the required amount of computation is reduced and the bandwidth of the transmitted data is improved.

[0008] According to the technical concept of the present disclosure, the application processor can minimize the use of hardware and layout area of ​​the image signal processing device as the operation of converting the pattern of an input image is offloaded to the neural network processing device. In addition, the image signal processing device can compensate for data loss areas that may occur due to conversion and improve image quality by correcting the first image based on the second image.

[0009] An application processor according to the technical concept of the present disclosure can provide a structure optimized for multimedia signal processing by controlling the on / off of an image signal processing module included in an image signal processing device according to the format of input data (e.g., image or video). Brief explanation of the drawing

[0010] FIG. 1 is a block diagram schematically illustrating an image processing system according to an exemplary embodiment of the present disclosure. FIG. 2 is a block diagram schematically illustrating an image processing system according to an exemplary embodiment of the present disclosure. FIG. 3 is a block diagram schematically illustrating image signal processing in an application processor according to an exemplary embodiment of the present disclosure. FIG. 4 is a block diagram schematically illustrating image signal processing in an application processor according to an exemplary embodiment of the present disclosure. FIG. 5 is a block diagram schematically illustrating image signal processing in an image signal processing device according to an exemplary embodiment of the present disclosure. FIG. 6 is a flowchart illustrating a method for training a neural network in a neural network processing device according to an exemplary embodiment of the present disclosure. FIG. 7 is a block diagram showing the configuration of a neural network according to an exemplary embodiment of the present disclosure. FIG. 8 shows an image processing system according to an exemplary embodiment of the present disclosure. FIG. 9 shows an image processing system according to an exemplary embodiment of the present disclosure. FIG. 10 is a block diagram of an electronic device including a multi-camera module to which an exemplary embodiment of the present disclosure may be applied. Specific details for implementing the invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0012] FIG. 1 is a block diagram schematically illustrating an image processing system according to an exemplary embodiment of the present disclosure.

[0013] Referring to FIG. 1, an image signal processing system (10) may include an image sensor (100) and an application processor (30, Application Processor; AP) that processes an image received from the image sensor (100).

[0014] An image sensor (100) can convert an optical signal incident through an optical lens into an electrical signal. The image sensor (100) may include a pixel array (not shown) comprising a plurality of pixels arranged in a matrix, and may include a color filter array (not shown) positioned above the pixel array. An image containing color information can be generated by an optical signal of a specific wavelength band that passes through the color filter array among the optical signals incident through the optical lens being incident on the pixel array.

[0015] The image sensor (100) can be mounted on an electronic device having an image or light sensing function. For example, the image sensor (100) can be mounted on an electronic device such as a camera, smartphone, wearable device, Internet of Things (IoT), tablet PC (Personal Computer), PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), navigation device, etc. Additionally, the image sensor (100) can be mounted on an electronic device equipped as a component in a vehicle, furniture, manufacturing equipment, door, various measuring instruments, etc.

[0016] The image sensor (100) can generate images of various patterns due to the arrangement of a color filter array, the difference in exposure time between multiple pixels, the position of the camera module (e.g., when the camera module is located at the bottom of the display), etc. Images of various patterns generated by the image sensor (100) can be provided to the application processor (30) as input images (IMG_IN).

[0017] The application processor (30) can perform image signal processing for an input image (IMG_IN) received from an image sensor (100). The application processor (30) can control the overall operation of image signal processing and may include a plurality of processing units that convert or process the input image (IMG_IN). The image converted or processed by the plurality of processing units may be provided to a display device (not shown). For example, the application processor (30) may be implemented as a System on Chip (SoC) in which a plurality of processing units are mounted on a single chip.

[0018] The application processor (30) may include a neural network processing unit (200) that performs a conversion on an input image (IMG_IN). The neural network processing unit (200) may convert the input image (IMG_IN) into a first image (IMG_1) of a first pattern using a neural network. The first pattern may refer to a pattern of an image processed by an image signal processing unit (300), and the input image (IMG_IN) may be an image of a pattern different from the first pattern, and conversion to the first pattern may be required.

[0019] The neural network processing device (200) can generate a second image (IMG_2) from an input image (IMG_IN) using a neural network. The second image (IMG_2) may include information to compensate for data loss areas that may occur as the pattern of the input image (IMG_IN) is transformed. The image signal processing device (300) can generate an image having a quality similar to the input image (IMG_IN) by correcting the first image (IMG_1) based on the second image (IMG_2).

[0020] The image signal processing device (300) can receive a first image (IMG_1) of a first pattern from the neural network processing device (200) and perform various image processing operations. For example, the image signal processing device (300) can perform at least one image processing among bad pixel correction, offset correction, lens distortion correction, phase detection auto focus, dynamic range compression, color gain correction, shading correction, gamma correction, denoising, and sharpening.

[0021] In some embodiments, the image signal processing device (300) may have a plurality of pipelines (320, 340). A pipeline may be applied in a pipeline method in which output data from a previous stage is used as input data for the next stage. A pipeline may refer to a configuration in which functional modules for processing data are connected. The image signal processing device (300) may include a first pipeline (320) and a second pipeline (340). The first pipeline (320) may receive a first image (IMG_1), and the second pipeline (340) may receive an image output from the first pipeline (320) and a second image (IMG_2), thereby performing image signal processing in stages.

[0022] According to an exemplary embodiment of the present disclosure, as the operation of converting the pattern of an input image (IMG_IN) is offloaded to a neural network processing device (200), the image signal processing device (300) can minimize the use of hardware and the layout area. Additionally, the image signal processing device (300) can compensate for data loss areas that may occur due to conversion and improve image quality by correcting the first image (IMG_1) based on the second image (IMG_2).

[0023] FIG. 2 is a block diagram schematically illustrating an image processing system according to an exemplary embodiment of the present disclosure.

[0024] Referring to FIG. 2, the image processing system (10a) may include an image sensor module (20a) including a neural network processing device (200a), and an application processor (30a) that processes an image received from the image sensor module (20a).

[0025] A neural network processing device (200a) according to an exemplary embodiment of the present disclosure may be composed of a low-complexity network that converts an input image (IMG_IN) provided from an image sensor (100a) into a first image (IMG_1) of a first pattern and generates a second image (IMG_2). Accordingly, the amount of computation required by the neural network processing device (200a) may be reduced and the bandwidth of the transmitted data may be improved. Due to the lightweighting of the neural network processing device (200a), the neural network processing device (200a) may be implemented by being mounted on an image sensor module (20a). The neural network processing device (200a) included in the image sensor module (20a) may be referred to in various ways, such as a neural processor, a neural engine, etc.

[0026] The neural network processing unit (200a) can be implemented in a stacked form with the sensor chip constituting the image sensor (100a). For example, the image sensor (100a) may be composed of a first chip, and the neural network processing unit (200a) may be composed of a second chip along with control logic, so that the two chips are stacked. However, the present invention is not limited thereto, and the image sensor (100a) and the neural network processing unit (200a) may be integrated on different semiconductors and electrically connected through a connecting member, or electrically connected to each other through a through-via.

[0027] The application processor (30a) can receive a first image (IMG_1) and a second image (IMG_2) from an image sensor module (20a). The application processor (30a) may further include a memory device (250a) to increase the image transmission speed. For example, the memory device may be a dynamic random access memory (DRAM) such as DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory), LPDDR (Low Power Double Data Rate) SDRAM, GDDR (Graphics Double Data Rate) SDRAM, RDRAM (Rambus Dynamic Random Access Memory), etc.

[0028] The memory device (250a) can receive and store at least one of a first image (IMG_1) and a second image (IMG_2) from the image sensor module (20a). The memory device (250a) can provide at least one of the first image (IMG_1) and the second image (IMG_2) to an image signal processing device (300a) in response to a command generated by an application processor (30a). In some embodiments, the memory device (250a) can provide the first image (IMG_1) to a first pipeline (310) and provide the second image (IMG_2) to a second pipeline (330).

[0029] FIG. 3 is a block diagram schematically illustrating image signal processing in an application processor according to an exemplary embodiment of the present disclosure. Fig. 3 will be described later with reference to Fig. 1.

[0030] Referring to FIG. 3, the image sensor (100) can generate images of various patterns according to the structure of the color filter array. The image sensor (100) may include a color filter array of a Bayer pattern. A Bayer pattern may refer to a pattern arranged in 2 rows and 2 columns and composed of a combination of a red filter, a blue filter, and two green filters.

[0031] Meanwhile, the image sensor (100) may include a color filter array having a pattern modified from a Bayer pattern. For example, the image sensor (100) may include a color filter array in which the green filter in the Bayer pattern is modified. The color filter array may be composed of a combination of a red filter, a blue filter, a green filter, and a white filter as one of the two green filters in the Bayer pattern is modified into a white filter. The color filter array may be composed of a combination of a red filter, a blue filter, and two white filters as each of the two green filters in the Bayer pattern is modified into a white filter. The white filter may be referred to in various ways, such as a clear filter, a transparent filter, etc.

[0032] The image sensor (100) may include a color filter array in which at least one of the color filters included in the Bayer pattern is transformed into one of a cyan filter, a magenta filter, and a yellow filter. For example, the color filter array may be composed of a combination of a red filter, a blue filter, and two yellow filters as two green filters in the Bayer pattern are each transformed into yellow filters. The color filter array may be composed of a combination of a cyan filter, a magenta filter, and two yellow filters. Additionally, the color filter array may be composed of a cyan filter, a magenta filter, a green filter, and a yellow filter. The image sensor (100) can generate images of various patterns depending on the arrangement of the color filter array.

[0033] In some embodiments, the image sensor (100) has a plurality of pixels, and the plurality of pixels may be divided into a plurality of adjacent pixels corresponding to the same color. The image sensor (100) may include a color filter array in which the same color filter is placed above each of the plurality of adjacent pixels. For example, the image sensor (100) may have a plurality of pixels divided into four adjacent pixels arranged in a 2x2 row and two columns corresponding to the same color. For example, the image sensor (100) may have a plurality of pixels divided into nine adjacent pixels arranged in a 3x3 row and three columns corresponding to the same color. For example, the image sensor (100) may have a plurality of pixels divided into sixteen adjacent pixels arranged in a 4x4 row and four columns corresponding to the same color. The image sensor (100) may generate images of various patterns according to the matrix arrangement of the plurality of pixels.

[0034] As illustrated in FIG. 3, the image sensor (100) has a plurality of pixels arranged in a matrix, and the plurality of pixels can be divided into four adjacent pixels arranged in 2 rows and 2 columns corresponding to the same color. For example, the image sensor (100) may have a structure in which the same color filter is placed on top of each of the four adjacent pixels arranged in 2 rows and 2 columns. Based on the structure of the color filter array, the image sensor (100) can generate an image in which a plurality of adjacent pixels represent the same color information. The image generated by the image sensor (100) may be provided to the neural network processing device (200) as an input image (IMG_IN). However, the present invention is not limited thereto, and the input image (IMG_IN) may have various patterns depending on the arrangement of the color filter array.

[0035] The neural network processing unit (200) can convert an input image (IMG_IN) into a first image (IMG_1) of a first pattern. In some embodiments, in response to the image signal processing unit (300) processing an image of a Bayer pattern, the neural network processing unit (200) can convert the input image (IMG_IN) into a first image (IMG_1) of a Bayer pattern. For example, four adjacent pixels corresponding to the same color in the input image (IMG_IN) can be converted into a combination of a green pixel (G), a red pixel (R), and two blue pixels (B). However, the present invention is not limited thereto, and the neural network processing unit (200) can convert the input image (IMG_IN) into an image of various patterns processed by the image signal processing unit (300).

[0036] The neural network processing unit (200) can generate a second image (IMG_2) from an input image (IMG_IN). The second image (IMG_2) may include information that compensates for a data loss region that may occur as the input image (IMG_IN) is converted into a first image (IMG_1) of a Bayer pattern. For example, the second image (IMG_2) may represent luminance information for the input image (IMG_IN).

[0037] In some embodiments, the image signal processing device (300) may have a plurality of pipelines. A first pipeline (320) may receive a first image (IMG_1) and perform image signal processing on the first image (IMG_1). A second pipeline (340) may receive a second image (IMG_2) and correct the first image (IMG_1). For example, the second pipeline (340) may apply filtering based on the luminance information of the second image (IMG_2) to the image output from the first pipeline (320). Accordingly, the image signal processing device (300) may generate an output image (IMG_OUT) having image quality similar to that of an input image (IMG_IN). An image signal processing device (300) according to an exemplary embodiment of the present disclosure can generate an output image (IMG_OUT) having a PSNR (Peak Signal-to-noise ratio) or SSIM (Structural similarity) with a value similar to that of an input image (IMG_IN).

[0038] FIG. 4 is a block diagram schematically illustrating image signal processing in an application processor according to an exemplary embodiment of the present disclosure. Fig. 4 will be described later with reference to Fig. 1.

[0039] Referring to FIG. 4, the image sensor (100) can generate images of various patterns according to a combination of multiple pixels with different exposure times. For example, the multiple pixels may be composed of a combination of pixels having a short exposure time that is exposed to a light source for a shorter time than a reference time, pixels having a long exposure time that is exposed to a light source for a longer time than a reference time, and pixels having a mid exposure time that is exposed to a light source for an average time.

[0040] As illustrated in FIG. 4, identical color filters are each placed on top of four adjacent pixels, and different exposure times may be set for the four adjacent pixels. For example, the four adjacent pixels may consist of one pixel (S) having a short exposure time, two pixels (M) having a medium exposure time, and one pixel (L) having a long exposure time. The image sensor (100) can generate an image including different dynamic ranges depending on the exposure times of the multiple pixels. The image generated by the image sensor (100) may be provided to the neural network processing device (200) as an input image (IMG_IN). However, the present invention is not limited thereto, and the input image (IMG_IN) may have various patterns depending on the difference in exposure time between the multiple pixels.

[0041] The neural network processing unit (200) can convert the input image (IMG_IN) into a first image (IMG_1) of a first pattern. In some embodiments, in response to the image signal processing unit (300) processing an image of a Bayer pattern, the neural network processing unit (200) can convert the input image (IMG_IN) into a first image (IMG_1) of a Bayer pattern. For example, four adjacent pixels with different dynamic ranges in the input image (IMG_IN) can be converted into a combination of green (G), red (R), and two blue (B) pixels. However, the present invention is not limited thereto, and the neural network processing unit (200) may convert the input image (IMG_IN) into an image of a different pattern processed by the image signal processing unit (300).

[0042] FIG. 5 is a block diagram schematically illustrating image signal processing in an image signal processing device according to an exemplary embodiment of the present disclosure.

[0043] Referring to FIG. 5, the image signal processing device (300) has a plurality of pipelines (320, 340), and each of the plurality of pipelines (320, 340) may include at least one image signal processing module.

[0044] In some embodiments, the image signal processing device (300) may include a multi-cell binning unit (MCB). The multi-cell binning unit (MCB) may be controlled to be on / off by the image signal processing device (300) according to the format of input data (e.g., image or video) received by the image signal processing device (300). For example, the multi-cell binning unit (MCB) may be turned off and not operate when the input data is an image, and turned on and operate when the input data is a video.

[0045] The image signal processing device (300) may include a first pipeline (320). The first pipeline (320) may receive a first image (IMG_1) and perform image signal processing. For example, the first pipeline (320) may include a bad pixel correction unit (BPC) and a noise removal unit (NR). The bad pixel correction unit (BPC) may correct defective pixels, such as dead pixels or hot pixels, that exist in the first image (IMG_1) due to the manufacturing process. The bad pixel correction unit (BPC) may correct the pixel values ​​corresponding to the bad pixels in the first image (IMG_1) to the pixel values ​​of pixels adjacent to the bad pixels. The noise removal unit (NR) may use a filter suitable for the type of noise (e.g., a Gaussian filter) to correct the pixel values ​​corresponding to the pixels containing noise to the pixel values ​​of pixels adjacent to the pixels containing noise.

[0046] The image signal processing device (300) may further include a second pipeline (340). The second pipeline (340) is connected to the downstream end of the first pipeline (320) and can perform image signal processing on an image output from the first pipeline (320). For example, the second pipeline (340) may include an RGB conversion unit (DMSC) and an image quality enhancement unit (IE). The RGB conversion unit (DMSC) can generate an RGB image in which each of multiple pixels contains all of the color information of red, green, and blue by performing color interpolation between adjacent pixels in the image output from the first pipeline (320). The image quality enhancement unit (IE) can apply filtering based on the second image (IMG_2) to the RGB image generated by the RGB conversion unit (DMSC). For example, the image quality enhancement unit (IE) can perform sharpening on the RGB image based on the luminance information represented by the second image (IMG_2).

[0047] However, the present invention is not limited thereto, and the first pipeline (320) and the second pipeline (340) may be configured with one of the aforementioned image signal processing modules omitted. Additionally, the first pipeline (320) and the second pipeline (340) may further include various image signal processing modules in addition to the aforementioned image signal processing modules.

[0048] FIG. 6 is a flowchart illustrating a method for training a neural network in a neural network processing device according to an exemplary embodiment of the present disclosure.

[0049] Referring to FIG. 6, a neural network processing device can generate a neural network composed of multiple parameters. For example, the neural network can be composed of various types of networks such as CNN (Convolutional Neural Network), R-CNN (Region with Convolutional Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc.

[0050] In step S110, the neural network processing unit may perform pre-processing on image data (e.g., training data). The neural network processing unit may increase the amount of limited image data by cropping pixels at arbitrary locations within the image data, rotating the image data at arbitrary angles, or flipping the image data. The neural network processing unit may vary the image data by adding noise (e.g., Gaussian random noise) to the image data.

[0051] In step S120, the neural network processing unit may obtain a dataset generated through data preprocessing in step S110. The dataset may be used as an original image representing the ground truth for training the neural network. For example, an original image of a Bayer pattern, an original image representing luminance information, and a noise-removed original image may be generated from the dataset. The dataset may be an RGB image in which each of multiple pixels contains all of the color information of red, green, and blue. An original image of a Bayer pattern may be generated by sampling color information corresponding to the format of the Bayer pattern from the RGB image. An original image representing luminance information may be generated by performing an operation to sample luminance information from the RGB image. Additionally, a noise-removed original image may be generated by applying a noise removal filter to the RGB image.

[0052] In step S130, the neural network processing unit can receive a dataset as input and perform training. The neural network can complete training once on the entire dataset based on the set parameters.

[0053] In step S140, the neural network processor uses a loss function (L JOINT The training of the neural network performed in step S130 can be evaluated based on the loss function (L JOINT ) can be used as an indicator to determine optimal parameters during the training process of a neural network. The loss function (L JOINT If ) does not satisfy the minimum value (S140-N), the parameters of the neural network are the loss function (L JOINT ) can be updated in response to the direction in which it decreases, and step S130 can be repeated. Loss function (L JOINT When ) satisfies the minimum value (S140-Y), the optimal parameters of the neural network are determined and training can be completed.

[0054] In step S150, the neural network processing unit can load an input image into the trained neural network. The input image may include images of various patterns due to changes in the arrangement of the color filter array or differences in exposure time between multiple pixels.

[0055] In step S160, the trained neural network can generate a prediction image from an input image of various patterns. In some embodiments, the neural network may be implemented as a simple network that converts the input image into a first image of a Bayer pattern and generates a second image to compensate for the conversion.

[0056] FIG. 7 is a block diagram showing the configuration of a neural network according to an exemplary embodiment of the present disclosure.

[0057] Referring to FIG. 7, the neural network may be composed of a plurality of sub-networks. The sub-networks can output an image corresponding to the feature map from the input image by learning a feature map between the input image and the predicted image output from the sub-network based on a loss function.

[0058] In some embodiments, the neural network may include at least one of a first sub-network (SN_1), a second sub-network (SN_2), and a third sub-network (SN_3). The first sub-network (SN_1) may be trained to perform noise removal on an input image (IMG_IN). The first sub-network (SN_1) may include a plurality of blocks (or a plurality of layers), and each of the plurality of blocks may use data output from a previous block as input data. The first sub-network (SN_1) predicts an image (P) from the input image (IMG_IN). NR Can generate ).

[0059] In some embodiments, the first sub-network (SN_1) may include a first convolution block (CONV), a residual block (Residual in Residual Dense Block; RRDB), a second convolution block (CONV), and a third convolution block (CONV). The first sub-network (SN_1) may include blocks connected by skip connections. For example, data output from the first convolution block (CONV) may be used as input data for the third convolution block (CONV) without passing through the residual block (RRDB) and the second convolution block (CONV). However, the present invention is not limited thereto, and some of the plurality of blocks constituting the first sub-network (SN_1) may be omitted. In addition, the first sub-network (SN_1) may further include a convolution layer, a pooling layer, an activation layer, and a fully-connected layer. The first sub-network (SN_1) has a loss function (L) as shown in Equation 1 below. NR It can be trained based on ). The first sub-network (SN_1) is the noise-removed original image (RGB gt ) and the predicted image (P) generated from the first sub-network (SN_1). NR It can be learned in a direction where the difference with ) decreases.

[0060]

[0061] The second sub-network (SN_2) is the image (P) output from the first sub-network (SN_1). NRIt can be trained to receive ) and convert it into an image of a Bayer pattern. For example, the second sub-network (SN_2) may include at least one convolution block. The second sub-network (SN_2) receives the image (P) output from the first sub-network (SN_1). NR Predicted image (P) from ) IMG_1 ) can be generated. The second sub-network (SN_2) has a loss function (L) as shown in Equation 2 below. IMG_1 It can be trained based on ). The second sub-network (SN_2) is the original image of the Bayer pattern (BAYER gt ) and the predicted image (P) generated from the second sub-network (SN_2). IMG_1 It can be learned in a direction where the difference with ) decreases.

[0062]

[0063] The third sub-network (SN_3) is the image (P) output from the first sub-network (SN_1). NR It can be trained to generate an image representing luminance information from ). For example, the third sub-network (SN_3) may include at least one convolution block. The third sub-network (SN_3) is an image (P) generated from the first sub-network (SN_1). NR Predicted image (P) from ) IMG_2 ) can be generated. The third sub-network (SN_3) has a loss function (L) as shown in Equation 3 below. IMG_2 It can be trained based on ). The third sub-network (SN_3) is the original image representing luminance information (luminance gt ) and the predicted image (P) generated from the third sub-network (SN_3). IMG_2 It can be learned in a direction where the difference with ) decreases.

[0064]

[0065] The neural network has a loss function (L) of the first sub-network (SN_1) as shown in Equation 4 below. NR ), loss function of the second sub-network (SN_2) (L IMG_1 ), loss function of the third sub-network (SN_3) (L IMG_2 Loss function (L) composed of JOINT It can be learned based on ). The neural network processor uses hyperparameters (λ) depending on the sub-network. NR or λ IMG_2 By adjusting ), the weights of the loss function applied in each sub-network can be adjusted.

[0066]

[0067] FIG. 8 shows an image processing system according to an exemplary embodiment of the present disclosure.

[0068] Referring to FIG. 8, the image processing system may include an image sensor (100b) and an application processor (30b) that processes an image received from the image sensor (100b). The application processor (30b) may change the operation mode by controlling the on / off of an image processing module according to the format of the input data (e.g., an image or video).

[0069] In some embodiments, the image sensor (100b) may generate an image for each frame that constitutes the image. The image for each frame generated by the image sensor (100b) may be provided to the application processor (30b) as an input image (FR_IN). The application processor (30b) may perform image signal processing on the input image (FR_IN) provided from the image sensor (100b). Due to the characteristics of the image that must be processed at a high speed, the application processor (30b) may turn on the multi-cell-binning unit (MCB).

[0070] A multi-cell-binning unit (MCB) included in an image signal processing device (300b) can receive an input image (FR_IN). In some embodiments, the multi-cell-binning unit (MCB) can convert the input image (FR_IN) into a first image (IMG_1) of a Bayer pattern by summing multiple pixel signals corresponding to the same color. Compared to the operation of the neural network processing device (200) described in FIGS. 1 to 6, which converts the input image (IMG_IN) based on a neural network, the delay in image processing can be reduced by the multi-cell-binning unit (MCB) performing a simple computation process.

[0071] The neural network processing unit (200c) can receive an input image (FR_IN) and generate a second image (IMG_2) representing luminance information. The operation of generating the second image (IMG_2) in the neural network processing unit (200c), the operation of converting the input image (IMG_IN) into a first image (IMG_1) of a Bayer pattern in the image signal processing unit (300b), and the operation of performing image signal processing for the first image (IMG_1) in the first pipeline (320b) can be performed in parallel. Accordingly, the second image (IMG_2) is provided to the second pipeline (340b) at the time of operation of the second pipeline (340b), thereby reducing the delay in image processing.

[0072] FIG. 9 shows an image processing system according to an exemplary embodiment of the present disclosure.

[0073] Referring to FIG. 9, the image sensor can generate an image for each frame constituting the image. The image for each frame generated by the image sensor can be provided as an input image (FR_IN). For example, the image sensor can generate an image in which four adjacent pixels represent the same color information corresponding to the structure of a color filter array. Multiple frame images constituting the image can be provided sequentially to a neural network processing device (200c) and an image signal processing device (300b).

[0074] In some embodiments, the first frame image (FR_1) may be provided to a multi-cell-binning unit (MCB) included in an image signal processing device (300b). The multi-cell-binning unit (310b) can generate a first image (IMG_1) of a Bayer pattern by summing a plurality of pixel signals corresponding to the same color information in the first frame image (FR_1). For example, a green pixel corresponding to a Bayer pattern can be generated by summing pixel signals output from each of the four pixels corresponding to the green (G) color in the first frame image (FR_1). A red pixel corresponding to a Bayer pattern can be generated by summing pixel signals output from each of the four pixels corresponding to the red (R) color in the first frame image (FR_1). A blue pixel corresponding to a Bayer pattern can be generated by summing pixel signals output from each of the four pixels corresponding to the blue (B) color in the first frame image (FR_1).

[0075] A first frame image (FR_1) may be provided to a neural network processing unit (200c). The neural network processing unit (200c) may generate a second image (IMG_2) representing luminance information for the first frame image (FR_1). The operation of generating the second image (IMG_2) in the neural network processing unit (200c), the operation of converting the first frame image (FR_1) into a first image (IMG_1) of a Bayer pattern, and the operation of performing image signal processing for the first image (IMG_1) in the first pipeline (320b) may be performed in parallel.

[0076] The second pipeline (340b) can generate an RGB image by performing color interpolation on the image output from the first pipeline (320b). The second pipeline (340b) can apply filtering based on the second image (IMG_2) to the generated RGB image. However, the present invention is not limited thereto, and the second pipeline (340b) may not perform some image signal processing. Alternatively, the second pipeline (340b) may further perform image restoration based on information between adjacent frames, or further perform conversion from an RGB image to a YUV image.

[0077] FIG. 10 is a block diagram of an electronic device including a multi-camera module to which an exemplary embodiment of the present disclosure may be applied.

[0078] Referring to FIG. 10, the electronic device (1000) may include a camera module group (1100), an application processor (1200), a PMIC (1300), and storage (1400).

[0079] The camera module group (1100) may include a plurality of camera modules (1100a, 1100b, 1100c). Although an embodiment in which three camera modules (1100a, 1100b, 1100c) are arranged is illustrated in the drawings, the embodiments are not limited thereto. In some embodiments, the camera module group (1100) may be modified to include only two camera modules or to include n camera modules (where n is a natural number greater than or equal to 4).

[0080] The image processing device (1210) may include a plurality of sub-image processors (1212a, 1212b, 1212c), an image generator (1214), and a camera module controller (1216).

[0081] The image processing device (1210) may include a plurality of sub-image processors (1212a, 1212b, 1212c) corresponding to the number of camera modules (1100a, 1100b, 1100c).

[0082] Image data generated from the camera module (1100a) may be provided to the sub-image processor (1212a) via the image signal line (ISLa), image data generated from the camera module (1100b) may be provided to the sub-image processor (1212b) via the image signal line (ISLb), and image data generated from the camera module (1100c) may be provided to the sub-image processor (1212c) via the image signal line (ISLc). Such image data transmission may be performed, for example, using a Camera Serial Interface (CSI) based on the Mobile Industry Processor Interface (MIPI), but is not limited thereto.

[0083] Meanwhile, in some embodiments, a single sub-image processor may be arranged to correspond to a plurality of camera modules. For example, the sub-image processor (1212a) and the sub-image processor (1212c) may not be implemented separately as illustrated, but rather integrated into a single sub-image processor, and image data provided from the camera module (1100a) and the camera module (1100c) may be selected through a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor. At this time, the sub-image processor (1212b) may not be integrated and may receive image data from the camera module (1100b).

[0084] Additionally, in some embodiments, image data generated from a camera module (1100a) may be provided to a sub-image processor (1212a) via an image signal line (ISLa), image data generated from a camera module (1100b) may be provided to a sub-image processor (1212b) via an image signal line (ISLb), and image data generated from a camera module (1100c) may be provided to a sub-image processor (1212c) via an image signal line (ISLc). Furthermore, image data processed by the sub-image processor (1212b) may be provided directly to an image generator (1214), but image data processed by the sub-image processor (1212a) and image data processed by the sub-image processor (1212c) may be provided to the image generator (1214) after one of them is selected through a selection element (e.g., a multiplexer).

[0085] Each sub-image processor (1212a, 1212b, 1212c) can perform image processing such as bad pixel correction, 3A adjustment (Auto-focus correction, Auto-white balance, Auto-exposure), noise reduction, sharpening, gamma control, and remosaic on image data provided from the camera module (1100a, 1100b, 1100c).

[0086] In some embodiments, remosaic signal processing may be performed in each camera module (1100a, 1100b, 1100c) and then provided to a sub-image processor (1212a, 1212b, 1212c).

[0087] Image data processed by each sub-image processor (1212a, 1212b, 1212c) can be provided to an image generator (1214). The image generator (1214) can generate an output image using image data provided from each sub-image processor (1212a, 1212b, 1212c) according to image generating information or a mode signal.

[0088] Specifically, the image generator (1214) can generate an output image by merging at least some of the image data generated from the sub-image processors (1212a, 1212b, 1212c) according to image generation information or a mode signal. Additionally, the image generator (1214) can generate an output image by selecting any one of the image data generated from the sub-image processors (1212a, 1212b, 1212c) according to image generation information or a mode signal.

[0089] In some embodiments, the image generation information may include a zoom signal (or zoom factor). Additionally, in some embodiments, the mode signal may be a signal based, for example, on a mode selected by a user.

[0090] When the image generation information is a zoom signal (zoom factor) and each camera module (1100a, 1100b, 1100c) has a different viewing angle (angle of view), the image generator (1214) can perform different operations depending on the type of zoom signal. For example, when the zoom signal is a first signal, among the image data output from the sub-image processor (1212a) and the image data output from the sub-image processor (1212c), the image data output from the sub-image processor (1212a) and the image data output from the sub-image processor (1212b) can be used to generate an output image. If the zoom signal is a second signal different from the first signal, the image generator (1214) can generate an output image by using the image data output from the sub-image processor (1212c) and the image data output from the sub-image processor (1212b), among the image data output from the sub-image processor (1212a) and the image data output from the sub-image processor (1212c). If the zoom signal is a third signal different from the first and second signals, the image generator (1214) can generate an output image by selecting any one of the image data output from each sub-image processor (1212a, 1212b, 1212c) without performing such image data merging. However, the embodiments are not limited thereto, and the method of processing image data can be modified as needed.

[0091] Here, image processing such as demosaicing, downscaling to a video / preview resolution size, gamma correction, and HDR (High Dynamic Range) processing is performed in advance in sub-image processors (1212a, 1212b, 1212c), and then the processed image data is transmitted to the image generator (1214). Therefore, even if the processed image data is provided to the image generator (1214) as a single signal line through the selection unit (1213), the image merging operation of the image generator (1214) can be performed at high speed.

[0092] In some embodiments, the image generator (1214) receives multiple image data with different exposure times from at least one of a plurality of sub-image processors (1212a, 1212b, 1212c) and performs HDR (high dynamic range) processing on the multiple image data to generate merged image data with increased dynamic range.

[0093] The camera module controller (1216) can provide control signals to each camera module (1100a, 1100b, 1100c). The control signals generated from the camera module controller (1216) can be provided to the corresponding camera modules (1100a, 1100b, 1100c) through separate control signal lines (CSLa, CSLb, CSLc).

[0094] One of the plurality of camera modules (1100a, 1100b, 1100c) may be designated as a master camera (e.g., 1100b) according to image generation information including a zoom signal or a mode signal, and the remaining camera modules (e.g., 1100a, 1100c) may be designated as slave cameras. This information may be included in a control signal and provided to the corresponding camera modules (1100a, 1100b, 1100c) through separate control signal lines (CSLa, CSLb, CSLc).

[0095] The camera module operating as a master and slave may be changed according to the zoom factor or operation mode signal. For example, if the field of view of the camera module (1100a) is wider than the field of view of the camera module (1100b) and the zoom factor indicates a low zoom magnification, the camera module (1100a) may operate as a master and the camera module (1100b) may operate as a slave. Conversely, if the zoom factor indicates a high zoom magnification, the camera module (1100b) may operate as a master and the camera module (1100a) may operate as a slave.

[0096] In some embodiments, the control signal provided from the camera module controller (1216) to each camera module (1100a, 1100b, 1100c) may include a sync enable signal. For example, if the camera module (1100b) is a master camera and the camera modules (1100a, 1100c) are slave cameras, the camera module controller (1216) may transmit a sync enable signal to the camera module (1100b). The camera module (1100b) that receives this sync enable signal may generate a sync signal based on the received sync enable signal and provide the generated sync signal to the camera modules (1100a, 1100c) through a sync signal line (SSL). The camera module (1100b) and camera modules (1100a, 1100c) can be synchronized with this sync signal to transmit image data to the application processor (1200).

[0097] In some embodiments, a control signal provided from a camera module controller (1216) to a plurality of camera modules (1100a, 1100b, 1100c) may include mode information according to a mode signal. Based on this mode information, the plurality of camera modules (1100a, 1100b, 1100c) may operate in a first operation mode and a second operation mode with respect to the sensing speed.

[0098] A plurality of camera modules (1100a, 1100b, 1100c) can, in a first operating mode, generate an image signal at a first speed (e.g., generate an image signal at a first frame rate) and encode it at a second speed higher than the first speed (e.g., encode an image signal at a second frame rate higher than the first frame rate), and transmit the encoded image signal to an application processor (1200). At this time, the second speed may be 30 times or less of the first speed.

[0099] The application processor (1200) stores the received image signal, that is, the encoded image signal, in a memory (1230) provided internally or in a storage (1400) outside the application processor (1200), and subsequently reads the encoded image signal from the memory (1230) or the storage (1400) to decode it, and can display image data generated based on the decoded image signal. For example, a corresponding sub-image processor among a plurality of sub-image processors (1212a, 1212b, 1212c) of the image processing device (1210) can perform decoding, and can also perform image processing on the decoded image signal.

[0100] A plurality of camera modules (1100a, 1100b, 1100c) can generate an image signal at a third speed lower than a first speed in a second operation mode (e.g., generate an image signal at a third frame rate lower than a first frame rate) and transmit the image signal to an application processor (1200). The image signal provided to the application processor (1200) may be an unencoded signal. The application processor (1200) may perform image processing on the received image signal or store the image signal in memory (1230) or storage (1400).

[0101] The PMIC (1300) can supply power, such as power voltage, to each of the plurality of camera modules (1100a, 1100b, 1100c). For example, the PMIC (1300) can supply first power to the camera module (1100a) through a power signal line (PSLa), supply second power to the camera module (1100b) through a power signal line (PSLb), and supply third power to the camera module (1100c) through a power signal line (PSLc), under the control of the application processor (1200).

[0102] The PMIC (1300) can generate power corresponding to each of the plurality of camera modules (1100a, 1100b, 1100c) and adjust the power level in response to a power control signal (PCON) from the application processor (1200). The power control signal (PCON) may include power adjustment signals for each operating mode of the plurality of camera modules (1100a, 1100b, 1100c). For example, the operating mode may include a low power mode, and in this case, the power control signal (PCON) may include information about the camera module operating in the low power mode and the power level being set. The power levels provided to each of the plurality of camera modules (1100a, 1100b, 1100c) may be the same or different from each other. Additionally, the power level may be changed dynamically.

[0103] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. The true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 An application processor comprising: a neural network processing device that uses a neural network to convert an input image of a second pattern different from a first pattern provided by an image sensor into a first image of the first pattern and generates a second image having luminance information of the input image to compensate for the conversion; and an image signal processing device having a plurality of pipelines that respectively receive the first image of the first pattern and the second image of the first pattern from the neural network processing device and perform image signal processing, wherein the plurality of pipelines includes a first pipeline that performs image signal processing for the first image and a second pipeline connected to the downstream end of the first pipeline that generates a third image having an RGB format by performing color interpolation on the image output from the first pipeline and applies filtering based on the second image to the third image. Claim 2 An application processor according to claim 1, characterized in that the first pattern is a Bayer pattern. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 An application processor according to claim 1, wherein the input image is provided from an image sensor having a plurality of pixels, which are divided into four adjacent pixels corresponding to the same color. Claim 7 An application processor according to claim 1, wherein the neural network includes a sub-network, and the sub-network outputs an image corresponding to the feature map from the input image by learning a feature map between the input image and the predicted image output from the sub-network based on a loss function. Claim 8 An application processor according to claim 7, wherein the sub-network is trained based on a loss function including a difference value between the original image of the first pattern and the predicted image, and the trained sub-network outputs the first image from the input image. Claim 9 An application processor according to claim 7, wherein the sub-network is trained based on a loss function including a difference value between an original image representing luminance information and a predicted image, and the trained sub-network outputs the second image from the input image. Claim 10 An application processor according to claim 7, wherein the sub-network is trained based on a loss function including a difference value between a noise-removed original image and a predicted image, and the trained sub-network outputs a noise-removed image from the input image.

Images