Method and apparatus with image sensor signal processing

Abstract

A method and apparatus with image sensor signal processing are provided. A processor-implemented method includes obtaining a color filter array (CFA) input image, obtaining pattern information corresponding to the CFA, pre-processing the input image based on the pattern information, generating an inferred image by inputting the pre-processed input image to an artificial neural network (ANN) model, and outputting the inferred image based on the pattern information. An output image is generated by selecting, for each pixel, from any one of the preprocessed input image and the inferred image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0169917, filed on Nov. 29, 2023, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a method and apparatus having image sensor signal processing. Background Art

[0003] Ubiquitous computing allows a computer system to be used at any time and anywhere. Such computing can be implemented using portable electronic devices such as mobile phones, digital cameras, laptop computers, etc.

[0004] Specifically, an image capturing device may be equipped with an image sensor such as a camera and a camcorder. The image capturing device may capture images and record them on a recording medium to play them at any time. However, such portable electronic devices and / or image capturing devices may not achieve high performance, may be large in size, may be heavy in weight, and / or may consume a large amount of power. Summary of the Invention

[0005] This Summary of the Invention is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0006] In one or more general aspects, a processor-implemented method includes: obtaining a color filter array (CFA) input image; obtaining pattern information corresponding to the CFA; preprocessing the input image based on the pattern information; generating an inference image by inputting the preprocessed input image into an artificial neural network (ANN) model; and generating an output image by selecting, for each pixel, from either the preprocessed input image or the inference image based on the pattern information.

[0007] The step of generating the output image may include: determining a pixel value of a pixel having the same color as the color of a target pixel based on the preprocessed input image; and determining a pixel value of a pixel having a color different from the color of the target pixel based on the inference image.

[0008] The step of performing preprocessing may include: classifying the input image according to the color types of the CFA based on the pattern information.

[0009] The step of performing preprocessing may further include: extracting features corresponding to the input image for each color type of the CFA based on the pattern information.

[0010] The step of performing preprocessing may further include: downsampling the features.

[0011] There may be an ANN model for each color type of the CFA, and the step of generating the inferred image may include: generating the inferred image by inputting the preprocessed input image into the ANN model corresponding to each color type of the CFA.

[0012] The pattern information may include any one or both of: the pixel position information of the CFA associated with the pixel position and the color information based on the pixel position.

[0013] The color information may include any combination of any one, any two, or more of: the color channel information, the gain information, and the parallax information according to the pixel position.

[0014] The method may include: post-processing the output image based on the pattern information.

[0015] The step of performing post-processing may include: performing any one or both of denoising or super-resolution on the output image based on the pattern information.

[0016] The CFA input image may include any combination of any one, any two, or more of: a Bayer pattern image, a Tetra (or four-in-one pixel, pixel four-in-one) pattern image, and a nona (or nine-in-one pixel, pixel nine-in-one) pattern image.

[0017] The step of generating the inferred image may include: generating red, green, and blue (RGB) images corresponding to the preprocessed input image.

[0018] In one or more general aspects, a non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, configure the one or more processors to perform any one, any combination, or all of the operations and / or methods disclosed herein.

[0019] In one or more general aspects, a processor-implemented method includes: obtaining a color filter array (CFA) input image; obtaining pattern information corresponding to the CFA; preprocessing the input image based on the pattern information; generating an inferred image based on the preprocessed input image; and generating an output image by post-processing the inferred image based on the pattern information, wherein the pattern information may include any one or both of the pixel position information of the CFA associated with the pixel position and the color information based on the pixel position.

[0020] The steps of generating an inferred image may include: generating a super-resolution image corresponding to the input image by inputting the preprocessed input image into an artificial neural network (ANN) model.

[0021] The steps of generating an inferred image may include: generating a mosaic rearrangement image corresponding to the input image by inputting the preprocessed input image into the ANN model.

[0022] The steps of generating an inferred image may include: generating a denoised image by inputting the preprocessed input image into the ANN model, in which noise is removed from the input image.

[0023] In one or more general aspects, an electronic device includes: an image sensor combined with a color filter array (CFA), where the image sensor is configured to generate a CFA input image by sensing light that has passed through the CFA; and one or more processors configured to: obtain pattern information corresponding to the CFA; preprocess the input image based on the pattern information; generate an inferred image by inputting the preprocessed input image into an artificial neural network (ANN) model; and generate an output image by selecting, for each pixel, from either the preprocessed input image or the inferred image based on the pattern information.

[0024] To generate the output image, the one or more processors may be configured to: determine the pixel values of pixels having the same color as the color of the target pixel based on the preprocessed input image; and determine the pixel values of pixels having a color different from the color of the target pixel based on the inferred image.

[0025] To perform preprocessing, the one or more processors may be configured to: classify the input image according to the color types of the CFA based on the pattern information.

[0026] To perform preprocessing, the one or more processors may be configured to: extract features corresponding to the input image for each color type of the CFA based on the pattern information.

[0027] To perform preprocessing, the one or more processors are configured to downsample the features.

[0028] There may be an ANN model for each color type of the CFA, and to generate the inferred image, the one or more processors may be configured to: generate the inferred image by inputting the preprocessed input image into the ANN model corresponding to each color type of the CFA.

[0029] The pattern information may include any one or both of pixel position information of the CFA associated with the pixel position and color information based on the pixel position.

[0030] The color information may include any combination of any one, any two, or more of color channel information, gain information, and parallax information according to the pixel position.

[0031] The one or more processors may be configured to post-process the output image based on the pattern information.

[0032] For post-processing, the one or more processors may be configured to perform any one or both of denoising and super-resolution on the output image based on the pattern information.

[0033] The CFA input image may include any combination of any one, any two, or more of a Bayer pattern image, a four-pixel-in-one pattern image, and a nine-pixel-in-one pattern image.

[0034] To generate an inference image, the one or more processors may be configured to generate red, green, and blue (RGB) images corresponding to the preprocessed input image.

[0035] In one or more general aspects, an electronic device includes: one or more processors configured to obtain a color filter array (CFA) input image; obtain pattern information corresponding to the CFA; preprocess the input image based on the pattern information; generate an inference image based on the preprocessed input image; and generate an output image by post-processing the inference image based on the pattern information, where the pattern information may include any one or both of pixel position information of the CFA associated with the pixel position and color information based on the pixel position.

[0036] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 An example of an imaging system (or image capture system) according to one or more example embodiments is shown.

[0038] Figure 2 An example of performing a demosaicing method is shown.

[0039] Figure 3A and Figure 3B A comparison between a demosaicing method according to one or more example embodiments and a typical demosaicing method is shown.

[0040] Figures 4 to 6 Shows a demosaicing method according to one or more example embodiments.

[0041] Figure 7 Shows a demosaicing method according to one or more example embodiments.

[0042] Figure 8 Shows a signal processing method according to one or more example embodiments.

[0043] Figure 9 Shows an example of the configuration of an electronic device according to one or more example embodiments.

[0044] Throughout the drawings and the detailed description, unless otherwise described or provided, the same or similar reference numerals will be understood to represent the same or similar elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, proportions, and depictions of the elements in the drawings may be exaggerated. Detailed Description

[0045] The following detailed description is provided to assist the reader in obtaining a comprehensive understanding of the methods, devices, and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent. For example, the order of operations within the operations described herein and / or the order of the operations described herein are merely examples and are not limited to those set forth herein, but may be changed as will be apparent after understanding the disclosure of the present application, except for the order of operations that must occur in a specific order and / or except for the order within the operations that must occur in a specific order. As another example, the order of operations and / or the order within the operations may be performed in parallel, except for at least a portion of the order of operations that must occur in a sequence (e.g., a specific sequence) and / or the order within the operations that must occur in a sequence (e.g., a specific sequence). In addition, for greater clarity and conciseness, the description of known features may be omitted after understanding the disclosure of the present application.

[0046] Although terms such as "first", "second", and "third", or A, B, (a), (b), etc. may be used herein to describe various components, components, regions, layers, or parts, these components, components, regions, layers, or parts should not be limited by these terms. For example, each of these terms is not used to define the nature, order, or sequence of the corresponding component, component, region, layer, or part, but is only used to distinguish the corresponding component, component, region, layer, or part from other components, components, regions, layers, or parts. Thus, without departing from the teachings of the examples, the first component, the first component, the first region, the first layer, or the first part referred to in the examples described herein may also be referred to as the second component, the second component, the second region, the second layer, or the second part.

[0047] Throughout the specification, when a component or element is described as being "on", "connected to", "coupled to", or "joined to" another component, element, or layer, the component, element, or layer can be directly (e.g., in contact with the other component or element) "on", "connected to", "coupled to", or "joined to" the other component, element, or layer, or there can reasonably be one or more other components, elements, or layers therebetween. When a component or element is described as being "directly on", "directly connected to", "directly coupled to", or "directly joined to" another component, element, or layer, there are no other components, elements, or layers therebetween. Similarly, phrases such as "between" and "immediately between" and "adjacent to" and "immediately adjacent to" can be interpreted as described in the foregoing.

[0048] The terms used herein are for the purpose of describing various examples only and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. As non-limiting examples, the terms "comprises", "comprising", and "having" indicate the presence of stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof, or optionally the presence of alternative stated features, quantities, operations, components, elements, and / or combinations thereof. Further, while one embodiment may state that the terms "comprises", "comprising", and "having" indicate the presence of stated features, quantities, operations, components, elements, and / or combinations thereof, other embodiments may have one or more of the stated features, quantities, operations, components, elements, and / or combinations thereof absent.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, based on an understanding of the present application's disclosure. Unless explicitly defined herein otherwise, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present application's disclosure, and should not be interpreted in an idealized or overly formal sense. The use herein of the term "can" with respect to an example or embodiment (e.g., with respect to what an example or embodiment can include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, but not all examples are so limited.

[0050] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more thereof. Unless the corresponding description and examples require such a list (e.g., "at least one of A, B, and C") to be interpreted as having a conjunctive meaning, the phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. are intended to have a disjunctive meaning, and these phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. also include examples where there may be one or more of each of A, B, and / or C (e.g., any combination of one or more of each of A, B, and C).

[0051] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples described herein are provided only to illustrate some of the possible ways of implementing the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of the present application. The terms "example" or "embodiment" used herein have the same meaning (e.g., the phrase "in one example" has the same meaning as "in one embodiment", and "one or more examples" has the same meaning as "in one or more embodiments").

[0052] The example embodiments described herein can be implemented in various types of products (such as, by way of example, a personal computer (PC), a laptop computer, a tablet computer, a smart phone, a television (TV), a smart household appliance, a smart vehicle, a self-service terminal, and a wearable device). Hereinafter, the example embodiments will be described in detail with reference to the drawings. When describing the example embodiments with reference to the drawings, the same reference numerals denote the same components, and the repetitive descriptions related thereto will be omitted.

[0053] Figure 1 An example of an imaging system (also referred to herein as an image capture system) according to one or more example embodiments is shown.

[0054] Referring to Figure 1 , according to one or more example embodiments, the imaging system 100 may include a part of a mobile device (such as a smart phone, a tablet personal computer (PC), a camera, or other device). The imaging system 100 may include a lens 105, a color filter array (CFA) 110, an image sensor 115, and a processor 120 (e.g., one or more processors). However, not all of the components shown are necessary. The imaging system 100 can be implemented with more components than those shown, or can be implemented with fewer components than those shown. For example, the imaging system 100 may further include a controller, a memory, and a display.

[0055] One or more of the components of the imaging system 100 (e.g., one or more of the CFA 110, the image sensor 115, and the processor 120) may be instantiated as elements of a single integrated system such as a system-on-chip (SOC) or other integrated system. Additionally, according to various example embodiments, the CFA 110 and the image sensor 115 may be integrated as shown in the case where the pixels of the CFA 110 are formed or disposed on the surface of the image sensor 115.

[0056] Light 101 may pass through the lens 105 and the CFA 110 and may then be sensed by the image sensor 115, and the image sensor 115 may use the received light to generate a CFA input image. The lens 105 may include any suitable lens including, as non-limiting examples, a rectilinear lens, a wide-field (or fish-eye) lens, a fixed-focal length lens, a zoom lens, a fixed-aperture lens, and / or a variable-aperture lens, etc. The image sensor 115 may include a complementary metal-oxide semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, and / or other suitable image sensors.

[0057] The image sensor 115 may use light-receiving elements to convert light into an electrical signal. In one example, in order to obtain color information in addition to the intensity of light, the image sensor 115 may receive light by specifying the intensity of visible light of a specific wavelength passing through the CFA 110 and verifying the intensity of light of the wavelength at the corresponding position, implementing the color conversion of human perception of color.

[0058] The processor 120 may be an image signal processor (ISP). The processor 120 may perform demosaicking (or “debayer”) for generating missing pixels of each color channel to generate a red, green, and blue (RGB) image from the input signal sampled by the CFA 110. Typical demosaicking may be performed by interpolation using the sampled data based on signal processing. However, in the case of a high-frequency image region having high-frequency characteristics, the interpolation performed using neighboring pixels may cause artifacts (such as moire) due to aliasing.

[0059] Machine learning techniques (such as, deep learning) can be applied to image processing and demosaicking, and compared with such typical ISP methods, machine learning techniques may have more desirable performance. Machine learning-based demosaicking can perform image processing using a convolutional filter bank structure (such as, U-NET and Residual Dense Block (RDB)). In one example, typical techniques can be used to generate all pixels through the same convolutional filter bank structure. In this example, the position information of the CFA may not be considered, so the configuration of adjacent pixels may change according to the pixel positions in the CFA. Specifically, for green (G) pixels and red (R) / blue (B) pixels, the distribution of adjacent pixels may vary greatly. According to one or more example embodiments, a target pixel can represent a pixel that is the target of demosaicking.

[0060] As will be described in detail below, compared with the typical techniques discussed above, the processor 120 of one or more example embodiments can perform demosaicking using the pattern information corresponding to the CFA 110.

[0061] Figure 2 An example of performing a demosaicking method is shown.

[0062] Refer to Figure 2 , pixels in an image sensor can sense the degree of black-and-white brightness (or "gray scale"). When a filter is combined with a pixel, the color value of the pixel can be generated. For example, using a CFA in an image sensor can apply color to the pixels.

[0063] The resulting image obtained by applying the CFA to the image sensor can be defined as a CFA pattern here. The input signal sampled by the CFA (e.g., input signal 210) can be referred to as a CFA input image, an input image, a CFA pattern, a CFA raw image, etc. here.

[0064] The CFA can be arranged in various patterns. The pattern arranged as shown in FIG. 211 can be referred to as a Bayer pattern, the pattern arranged as shown in FIG. 213 can be referred to as a Tetra (or four-in-one pixel, pixel four-in-one) pattern, and the pattern arranged as shown in FIG. 215 can be referred to as a nona (or nine-in-one pixel, pixel nine-in-one) pattern. Although for ease of description, the CFA is described as having a Bayer pattern here, it is not necessarily limited to this. According to the design, various patterns (such as, Tetra pattern and nona pattern) can also be used for the CFA.

[0065] A pixel can be combined with only one of the R filter, G filter, and B filter, and thus one image pixel of the image sensor 115 can sense only one color. To compensate for this, neighboring pixels can be used to artificially generate the missing colors in each pixel. For the actual image that people view, all R colors, G colors, and B colors can be viewed together, and this can also be the result of artificial generation by neighboring pixels, and such processing can be called demosaicking (or "debayer"). For example, demosaicking can be an operation of generating an RGB image 220 using the input signal 210 sampled by the CFA.

[0066] Figure 3A and Figure 3B shows a comparison between a demosaicking method according to one or more example embodiments and a typical demosaicking method.

[0067] Referring Figure 3A , a G image 330 can be generated by demosaicking the Bayer raw image 310 sampled by the CFA using a two-layer convolutional filter 320.

[0068] FIGS. 311, 312, and 313 respectively show cases where the convolutional filter is applied to the pixel position Loc0 311-1, the pixel position Loc1 311-2, and the pixel position Loc2 311-3. A convolution operation can be performed between the Bayer raw image 310 and the Conv0 filter of the convolutional filter 320, and a convolution operation can be performed between the result (e.g., the result of the convolution operation performed between the Bayer raw image 310 and the Conv0 filter) and the Conv1 filter of the convolutional filter 320. The expression "applying a convolutional filter" used herein can indicate performing a convolution operation based on the convolutional filter.

[0069] In the cases of Loc0 311-1 and Loc1 311-2, the weights of the convolutional filter can be applied to the same-color pixels, while in the case of Loc2 311-3, the weights of the two-layer convolutional filter 320 can be applied to color pixels different from the color pixels of Loc0 311-1 and Loc1 311-2. For example, in the cases of Loc0 311-1 and Loc1 311-2, the w 22 of the Conv0 layer can all be applied to the blue pixels, and then the w 11 of the Conv1 layer can be applied to the result that has passed through the Conv0 layer. On the other hand, in the case of Loc2 311-3, the w 22 of the Conv0 layer can be applied to the green pixels, and then the w 11It can be applied to the result that has passed through the Conv0 layer. For example, according to the color channels of the output positions, the colors of the pixels to which the weights of each convolutional filter are applied can vary.

[0070] As described above, when typical methods and apparatuses apply convolutional filters to the entire output without considering the CFA pattern, which may lead to differences in the data distribution between color pixels, typical methods and apparatuses may not be able to learn the optical filter weights, and when the same filter is applied to each pixel position, typical methods and apparatuses may result in unexpected outcomes.

[0071] Therefore, the data distribution can vary according to color pixels, and for optimal filter weights, the methods and apparatuses of one or more embodiments can perform learning on the same color pixels.

[0072] Referring to Figure 3B , image 331 is a synthetically generated green resolution map and is the target image to be restored, and pattern 333 is the Bayer pattern corresponding to the target image 331. For example, the target image 331 can be an image input to a CFA that only includes a green filter, and the Bayer pattern 333 can be a pattern generated when the target image 331 has passed through the CFA. When the target image 331 only includes green, the red pixel regions and blue pixel regions of the Bayer pattern 333 may not have values, and only the green pixel regions may have values. For example, in the target image 331, the parts shown in black can indicate the red pixel regions and blue pixel regions that do not have any values.

[0073] In one example, a typical method can generate a G image among the RGB images (i.e., a demosaicked image obtained by demosaicking all pixels to green) by using a first filter 341 and a second filter 343 for the target image 331, while the methods of one or more exemplary embodiments can generate an output image 361 of green among the RGB images by performing demosaicking on the target image 331.

[0074] When the first filter 341 and the second filter 343 are applied without considering the CFA pattern as in a typical method, a greater error may occur at the positions of the green pixels compared to the positions of the red and blue pixels. For example, as shown in the green image 351 generated by performing demosaicing by applying the first filter 341 and the second filter 343 to the target image 331, the pixel position 333-1 on the Bayer pattern 333 (and the same pixel position on the target image 331) may have green, but the pixel position 351-1 of the green image 351 at the same position as the pixel position 333-1 may have a value of zero "0". For example, in the green image 351, by applying the first filter 341 and the second filter 343, the pixel regions corresponding to the red pixel region and the blue pixel region shown in black in the target image 331 may have a non-zero (0) value, while by applying the first filter 341 and the second filter 343, the pixel region corresponding to the green region (e.g., pixel position 333-1) on the target image 331 may have a value of 0.

[0075] In a typical method, the error image 353 may be an image generated based on the difference between the target image 331 and the green image 351. Closer to black may indicate a smaller error, and closer to white may indicate a greater error. As shown in the error image 353, a greater error may occur at the positions of the green pixels (e.g., position 353-1) compared to the positions of the red and blue pixels.

[0076] According to one or more example embodiments described in further detail below, compared to a typical method, the methods and apparatuses of one or more embodiments may use the position information of the CFA pattern to apply a convolutional filter to the positions of the red and blue pixels, and may use the input image as it is for the positions of the green pixels. As shown in the output image 361 and the error image 363 obtained by performing demosaicing by the methods and apparatuses of one or more embodiments, the methods and apparatuses of one or more embodiments may recover the target image without significant error.

[0077] As described above, the distribution of neighboring pixels may vary according to the positions of the color pixels in the CFA, and thus when using a learning-based artificial neural network (ANN) model, the methods and apparatuses of one or more embodiments may consider the CFA pattern for effective recovery of the output image.

[0078] Specifically, in the case of a four-pixel binning pattern (e.g., 4×4) or a nine-pixel binning pattern (e.g., 9×9) other than the Bayer pattern (e.g., 2×2), the sampling interval between color channels can be widened by the CFA pattern, and the data distribution of adjacent pixels through the CFA pattern can be made more diverse. Therefore, it may be even more advantageous for the methods and apparatuses of one or more embodiments to consider the CFA pattern for demosaicking.

[0079] Figures 4 to 6 Shows a demosaicking method according to one or more example embodiments.

[0080] Referring to Figure 4 , the above reference to Figure 1 and Figure 2 The content described can be applied to the content described in the following reference to Figure 4 The content described above. The above reference to Figure 1 The processor 120 of the imaging system 100 described may include a CFA information transmission module 410, a preprocessing module 420, an ANN model 430, and an output processing module 440. The term "module" used in the CFA information transmission module 410, the preprocessing module 420, and the output processing module 440 may represent hardware (e.g., hardware implementing software and / or firmware), and may be used interchangeably with other terms (e.g., "logic", "logic block", "component", or "circuit"). A module can be a single integrated component configured to perform one or more functions or its smallest unit or part. A module can be implemented mechanically or electronically. A module may include, for example, at least one of an application specific integrated circuit (ASIC) chip, a field programmable gate array (FPGA), and a programmable logic device configured to perform a specific operation.

[0081] The CFA information transmission module 410 may send pattern information corresponding to the CFA to the preprocessing module 420 and the output processing module 440. The pattern information as "CFA information according to pixel position" may include, for example, RGB color channel information, gain values according to color pixels, and / or parallax information, etc.

[0082] The CFA input image may be input to the preprocessing module 420, and the preprocessing module 420 may receive the pattern information from the CFA information transmission module 410. The preprocessing module 420 may preprocess the CFA input image based on the pattern information.

[0083] Referring again to Figure 4 , according to another example embodiment, the preprocessing module 420 may send the unchanged CFA input image to the output processing module 440.

[0084] According to another exemplary embodiment, the preprocessing module 420 may extract features corresponding to the input image for each color type of the CFA through preprocessing based on the CFA pattern. In addition, in order to reduce the amount of calculation, the preprocessing module 420 may perform downsampling after the convolutional layer of the color filter for each pixel type.

[0085] The ANN model 430 may be a model trained to perform demosaicing and is not limited to a specific structure and type of network.

[0086] Referring to Figure 5 , the CFA input image 510 may be classified into a red input image 520-1, a green input image 520-2, and a blue input image 520-3 according to the color type of the CFA. The preprocessing module 420 may send the CFA input image 510 to the ANN model 430 and the output processing module 440, and / or send the red input image 520-1, the green input image 520-2, and the blue input image 520-3 classified according to the color type to the ANN model 430 and the output processing module 440.

[0087] The ANN model 430 may receive the red input image 520-1, the green input image 520-2, and the blue input image 520-3 classified according to the color type, and generate inference images 530-1, 530-2, and 530-3. The ANN model 430 may be trained to perform demosaicing.

[0088] In the case of a typical method, the network may be configured without considering the CFA, and it may be difficult for a typical method to respond to changes in the distribution of neighboring color pixels according to the target position. Therefore, the network of a typical method may not be optimally trained in the learning process through backpropagation.

[0089] In contrast, according to one or more exemplary embodiments, the ANN model 430 may generate an output considering the CFA, and thus the color pixel distribution of the pixels to which the filter weights are applied may be the same, and thus the methods and apparatuses of one or more embodiments may facilitate learning and high performance. For example, when the filter weight applied to red is applied to the red CFA input image 520-1, the ANN model 430 may have the same color pixel distribution.

[0090] According to one or more example embodiments, the inferred images 530-1, 530-2, and 530-3 may be RGB images generated by the ANN model 430 after inference training. To distinguish from the red R, green G, and blue B in the CFA input image 510 and the red input image 520-1, green input image 520-2, and blue input image 520-3, the red, green, and blue in the inferred images 530-1, 530-2, and 530-3 may be indicated as R', G', and B', respectively.

[0091] Referring to Figure 6 , according to one or more example embodiments, the processor 120 may use multiple ANN models. The processor 120 may use separate ANN models for each color to generate inferred images. For example, the R ANN model 610 may receive the red input image 520-1 to generate a red inferred image, the G ANN model 620 may receive the green input image 520-2 to generate a green inferred image, and the B ANN model 630 may receive the blue input image 520-3 to generate a blue inferred image.

[0092] Referring again to Figure 4 , according to one or more example embodiments, the output processing module 440 may generate an output image by selecting one between the preprocessed input image and the inferred image for each pixel based on the pattern information. The output processing module 440 may consider the CFA pattern and the position information of the output pixel to generate the output image. The output processing module 440 may use the CFA pattern and the position information of the output pixel to generate the output image based on the input image and / or the preprocessed data received from the preprocessing module 420 and the inferred image of the ANN model 430.

[0093] For example, the output processing module 440 may determine the pixel value of a pixel having the same color as the color of the target pixel based on the preprocessed input image, and may determine the pixel value of a pixel having a color different from the color of the target pixel (hereinafter also referred to as a "missing pixel") based on the inferred image.

[0094] Referring to Figure 5, the output processing module 440 can generate output images 540-1, 540-2, and 540-3 based on the red input image 520-1, the green input image 520-2, the blue input image 520-3, and the inference images 530-1, 530-2, and 530-3. For example, when the target pixel is red, the output processing module 440 can determine the pixel value of the first pixel 550-3 of "red having the same color as the color of the target pixel" as the pixel value R of the input image 550-1 corresponding to the first pixel 550-3. In contrast, the output processing module 440 can determine the pixel value of the second pixel 550-4 of "having a color different from the color of the target pixel (e.g., blue)" as the pixel value R' based on the inference image 530-1.

[0095] As described above, according to one or more example embodiments, the ANN model 430 can generate output considering the CFA, thereby facilitating effective learning of the filter weights of the ANN.

[0096] To verify it, experiments were conducted, and in each experiment, a residual dense network (RDN) was used as the ANN model. To verify the feasibility, the demosaicking results were checked from the green pixels.

[0097] Table 1 shows the peak signal-to-noise ratio (PSNR) performance according to the network size obtained from a typical demosaicking method and the demosaicking methods of one or more example embodiments.

[0098] Table 1:

[0099]

[0100] The Bayer structure is used as the structure of the CFA, and according to one or more example embodiments of the present disclosure, the imaging system is configured such that the preprocessing module sends the input image to the output processing module, the ANN model is configured as a single model, and the output processing module provides the output of the ANN model for the missing pixels. For example, FLOP can represent floating-point operations / second, and #Param can represent the number of parameters of the ANN model.

[0101] Referring to Table 1 above, it is verified that as the network size decreases, compared with the typical demosaicking method, the PSNR values obtained from the demosaicking methods of one or more example embodiments of the present disclosure increase by +0.143, +0.332, and +0.538.

[0102] Table 2:

[0103]

[0104] In addition, Table 2 shows the results of evaluating the PSNR performance according to the type of CFA. It is verified that compared with the typical demosaicing method, the PSNR values ​​obtained from the demosaicing method of one or more example embodiments of the present disclosure are increased by +0.178, +0.815, and +1.45 in the Bayer color filter, the four-pixel-in-one color filter, and the nine-pixel-in-one color filter, respectively. In the case where the CFA is a four-pixel-in-one color filter or a nine-pixel-in-one color filter, it is verified that the sampling interval between the same colors is longer than the sampling interval between the same colors of the Bayer color filter, and the color distribution of adjacent pixels is greatly changed. Therefore, it is verified that the effect of the present disclosure is further maximized.

[0105] In addition, as described above, Table 3 shows the result of performing preprocessing (eg, filtering and downsampling) in consideration of the CFA pattern by the preprocessing module to reduce the amount of calculation.

[0106] Table 3:

[0107]

[0108] The results obtained using the typical demosaicing method are compared with the results obtained using one or more example embodiments of the preprocessing module for reducing the amount of calculation, and it is verified that the demosaicing method of one or more example embodiments has the effect of reducing the amount of calculation by 73% while maintaining the advantage of 0.6dB PSNR compared to the typical demosaicing method. The demosaicing method of one or more example embodiments is highly suitable for real-time processing situations where delay reduction is important.

[0109] Figure 7 A demosaicing method according to one or more example embodiments is illustrated.

[0110] Reference Figure 7 For ease of description, operations 710 to 750 are described as using Figure 1 However, operations 710 to 750 may be performed by any other suitable electronic device and used in any suitable system.

[0111] In addition, the following operations can be performed as follows Figure 7 The operations are performed in the order and manner shown and described, but the order of one or more of the operations may be changed, one or more of the operations may be omitted, and / or two or more of the operations may be performed in parallel or simultaneously without departing from the spirit and scope of the example embodiments described herein.

[0112] The imaging system 100 of one or more example embodiments may obtain a CFA input image in operation 710. The CFA input image may include at least one of a Bayer pattern image, a four-pixel-in-one pattern image, and a nine-pixel-in-one pattern image.

[0113] For example, image sensor 115 may generate a CFA input image by sensing light that has passed through CFA 110 , and may transmit the generated CFA input image to processor 120 .

[0114] In operation 720, the imaging system 100 may obtain pattern information corresponding to the CFA. The pattern information may include at least one of pixel position information of the CFA and color information according to the pixel position. The color information may include at least one of color channel information, gain information, and disparity information according to the pixel position.

[0115] In operation 730, the imaging system 100 may pre-process the input image based on the pattern information. The imaging system 100 may classify the input image according to the color type of the CFA based on the pattern information. The imaging system 100 may also extract features corresponding to the input image for each color type of the CFA based on the pattern information. The imaging system 100 may downsample the features. In one example, the features corresponding to the input image or the downsampled features may be determined as the pre-processed input image.

[0116] In operation 740, the imaging system 100 may generate an inferred image by inputting the preprocessed input image to the ANN model. The imaging system 100 may input the preprocessed input image to the ANN model to generate an RGB image corresponding to the input image.

[0117] In operation 750, the imaging system 100 may generate an output image by selecting one between the preprocessed input image and the extrapolated image for each pixel based on the pattern information. The imaging system 100 may determine pixel values ​​of pixels having the same color as the target pixel based on the preprocessed input image, and may determine pixel values ​​of pixels having a color different from the target pixel based on the extrapolated image.

[0118] The imaging system 100 may perform post-processing on the output image based on the pattern information. The imaging system 100 may perform at least one of denoising and super-resolution on the output image based on the pattern information.

[0119] An ANN model may exist for each color type of the CFA, and the imaging system 100 may input the pre-processed input image to the ANN model corresponding to each color type of the CFA to generate an inferred image.

[0120] Figure 8A signal processing method according to one or more example embodiments is shown.

[0121] Referring Figure 8 , for ease of description, operations 810 to 850 are described as being performed using Figure 1 the imaging system 100 shown in

[0122] Furthermore, the following operations may be performed in the order and manner shown and described below with reference to Figure 8 However, without departing from the spirit and scope of the example embodiments described herein, the order of one or more of the operations may be changed, one or more of the operations may be omitted, and / or two or more of the operations may be performed in parallel or simultaneously.

[0123] In addition to the demosaicing method described above with reference to Figures 1 to 7 , the signal processing method of one or more example embodiments may be used for any module and task that can be processed in the raw domain sampled by the CFA. For example, the signal processing method may be used for denoising, remosaicing (or "re-mosaicking"), super-resolution, bad pixel correction, etc.

[0124] In operation 810, the imaging system 100 of one or more example embodiments may obtain a CFA input image.

[0125] In operation 820, the imaging system 100 may obtain pattern information corresponding to the CFA. The pattern information may include at least one of pixel position information of the CFA and color information according to the pixel position. The color information may include at least one of color channel information, gain information, and parallax information according to the pixel position.

[0126] In operation 830, the imaging system 100 may preprocess the input image based on the pattern information.

[0127] In operation 840, the imaging system 100 may generate an inference image by inputting the preprocessed input image into an ANN model. For example, the imaging system 100 may generate a super-resolution image corresponding to the input image by inputting the preprocessed input image into the ANN model. For example, the imaging system 100 may generate a mosaic image corresponding to the input image by inputting the preprocessed input image into the ANN model. For example, the imaging system 100 may generate a denoised image by inputting the preprocessed input image into the ANN model to remove noise from the preprocessed input image. The ANN model may be trained to perform tasks (such as, mosaic rearrangement, super-resolution, missing pixel correction, etc.). When the ANN model is trained to perform mosaic rearrangement, the ANN model may receive the pixel values of a bundle of pixels and convert them back to the original Bayer pattern.

[0128] In operation 850, the imaging system 100 may generate an output image by post-processing the inference image based on pattern information. For example, when the imaging system 100 performs mosaic rearrangement, the imaging system 100 may determine the pixel values of the pixels having the same color as the color of the target pixel based on a bundle of pixels, and determine the pixel values of the pixels having a color different from the color of the target pixel based on the inference image.

[0129] Figure 9 An example showing the configuration of an electronic device according to one or more example embodiments.

[0130] Referring to Figure 9 , the electronic device 900 may include a processor 910 (e.g., one or more processors), a memory 920 (e.g., one or more memories), a camera 930, a storage device 940, an input device 950, an output device 960, and a network interface 970, and these components may communicate with each other via a communication bus 980. For example, the electronic device 900 may be or be implemented as a mobile device (such as, a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, or a laptop computer), a wearable device (such as, a smart watch, a smart wristband, or smart glasses), or at least a part of a computing device (such as, a desktop computer or a server). The electronic device 900 may structurally and / or functionally include Figure 1 the imaging system 100.

[0131] The processor 910 may execute instructions and functions in the electronic device 900. For example, the processor 910 may process instructions stored in the memory 920 or the storage device 940. The processor 910 may execute the above-referred Figures 1 to 8One or more of the operations and methods described. Memory 920 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Memory 920 may store instructions to be executed by processor 910 and may also store information associated with software and / or applications when the software and / or applications are executed by electronic device 900. In one example, memory 920 may be or include a non-transitory computer-readable storage medium storing instructions that, when executed by processor 910, configure processor 910 to perform any one, any combination, or all of the operations and methods described herein with reference to Figures 1 to 8 Any one, any combination, or all of the operations and methods described. In one example, processor 910 may be or include Figure 1 Processor 120. In another example, processor 120 may be an ISP, and camera 930 may include processor 120.

[0132] Camera 930 may capture photos and / or videos. Camera 930 may include Figure 1 Lens 105, CFA 110, and image sensor 115. Storage device 940 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Storage device 940 may store a larger amount of information than memory 920 and store the information for a long period of time. For example, storage device 940 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other non-volatile memory.

[0133] Input device 950 may receive input from a user by using traditional input schemes such as a keyboard and a mouse and by new input schemes such as touch input, voice input, and image input. Input device 950 may include, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from a user and send the detected input to electronic device 900. Output device 960 may provide the output of electronic device 900 to the user through visual, auditory, or tactile channels. Output device 960 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. For example, output device 960 may provide an output image generated by processor 910 through a display. Network interface 970 may communicate with external devices through a wired or wireless network.

[0134] Imaging system, lens, CFA, image sensor, processor, CFA information transmission module, preprocessing module, output processing module, electronic device, memory, camera, storage device, input device, output device, network interface, imaging system 100, lens 105, CFA 110, image sensor 115, processor 120, CFA information transmission module 410, preprocessing module 420, output processing module 440, electronic device 900, processor 910, memory 920, camera 930, storage device 940, input device 950, output device 960, network interface 970 that can communicate with each other through the communication bus 980 described herein, and these components (including regarding Figures 1 to 9is implemented by or represents a hardware component. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include, where appropriate: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For the sake of brevity, the singular terms "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, can implement a single hardware component, or two or more hardware components. As described above, or in addition to the above description, example hardware components can have any one or more of different processing configurations, examples of different processing configurations include: single processor, independent processor, parallel processor, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0135] Figures 1 to 9 shown in and with respect to Figures 1 to 9The method of performing the operations described in this application is performed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions (e.g., computer or processor / processing device readable instructions) or software as described above to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

[0136] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the method as described above may be written as a computer program, code segment, instruction, or any combination thereof to individually or jointly direct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and method as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executable by one or more processors or computers. In another example, the instructions or software include high-level code executable by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the figures and the corresponding descriptions herein, which disclose algorithms for performing the operations performed by the hardware components and method as described above.

[0137] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and execute the methods described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed on one or more non-transitory computer-readable storage media, or recorded, stored, or fixed on one or more non-transitory computer-readable storage media. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDD), solid state drives (SSD), card memories (such as, multimedia cards or micro cards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid state disks, and / or any other device, any other device being configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers such that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0138] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of this application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered only as descriptive and not for purposes of limitation. The description of a feature or aspect in each example should be considered applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0139] Accordingly, in addition to what has been above and in all the appended drawings disclosed, the scope of the disclosure also includes the claims and their equivalents, that is, all variations within the scope of the claims and their equivalents will be construed as included in the disclosure.

Claims

1. A method for image sensor signal processing, comprising: obtaining a color filter array input image; obtaining pattern information corresponding to a color filter array; Preprocessing the color filter array input image based on the pattern information; generating an inferred image by inputting the preprocessed color filter array input image into an artificial neural network model; as well as Based on the pattern information, an output image is generated by selecting one from the pre-processed color filter array input image and the inferred image for each pixel.

2. The method according to claim 1, wherein: The steps to generate the output image are: determining pixel values ​​of pixels having the same color as the target pixel based on the preprocessed color filter array input image; and Based on the inferred image, pixel values ​​of pixels having a color different from the color of the target pixel are determined.

3. The method according to claim 1, wherein: The step of preprocessing includes classifying the color filter array input image according to the color type of the color filter array based on the pattern information.

4. The method according to claim 1, wherein: The step of performing preprocessing includes: extracting features corresponding to the color filter array input image for each color type of the color filter array based on the pattern information.

5. The method according to claim 4, wherein: The step of preprocessing also includes: downsampling the features.

6. The method according to claim 1, wherein There is an artificial neural network model for each color type of the color filter array, and The steps to generate the inferred image include: An inferred image is generated by inputting the preprocessed color filter array input image into an artificial neural network model corresponding to each color type of the color filter array.

7. The method according to claim 1, wherein: The pattern information includes any one or both of: pixel position information of a color filter array associated with a pixel position and color information based on the pixel position.

8. The method according to claim 7, wherein: The color information includes: any one or any combination of any two or more of the color channel information, gain information and disparity information according to the pixel position.

9. The method according to claim 1, further comprising: The output image is post-processed based on the pattern information.

10. The method according to claim 9, wherein: The step of performing post-processing includes: performing any one or both of denoising and super-resolution on the output image based on the pattern information.

11. The method according to any one of claims 1 to 10, wherein: The color filter array input image includes any one of a Bayer pattern image, a four-pixel-in-one pattern image, and a nine-pixel-in-one pattern image, or any combination of any two or more of them.

12. The method according to any one of claims 1 to 10, wherein: The step of generating an inferred image includes generating red, green and blue RGB images corresponding to the pre-processed color filter array input image.

13. A method for image sensor signal processing, comprising: obtaining a color filter array input image; obtaining pattern information corresponding to a color filter array; Preprocessing the color filter array input image based on the pattern information; generating an inferred image based on the preprocessed color filter array input image; as well as generating an output image by post-processing the inferred image based on the pattern information, The pattern information includes any one or both of pixel position information of a color filter array associated with a pixel position and color information based on the pixel position.

14. The method according to claim 13, wherein: The step of generating an inferred image includes generating a super-resolution image corresponding to the color filter array input image by inputting the preprocessed color filter array input image into an artificial neural network model.

15. The method according to claim 13, wherein: The step of generating an inferred image includes: generating a mosaic rearranged image corresponding to the color filter array input image by inputting the preprocessed color filter array input image into an artificial neural network model.

16. The method according to claim 13, wherein: The step of generating the inferred image includes generating a denoised image by inputting the pre-processed color filter array input image to an artificial neural network model to remove noise from the pre-processed color filter array input image.

17. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 16.

18. An electronic device comprising: an image sensor in combination with the color filter array, wherein the image sensor is configured to generate a color filter array input image by sensing light that has passed through the color filter array; and One or more processors configured to: obtaining pattern information corresponding to a color filter array; Preprocessing the color filter array input image based on the pattern information; generating an inferred image by inputting the preprocessed color filter array input image into an artificial neural network model; and Based on the pattern information, an output image is generated by selecting one from the pre-processed color filter array input image and the inferred image for each pixel.

19. The electronic device according to claim 18, wherein: To generate an output image, the one or more processors are configured to: determining pixel values ​​of pixels having the same color as the target pixel based on the preprocessed color filter array input image; as well as Based on the inferred image, pixel values ​​of pixels having a color different from the color of the target pixel are determined.

20. The electronic device according to claim 18, wherein: For pre-processing, the one or more processors are configured to classify the color filter array input image according to the color type of the color filter array based on the pattern information.

21. The electronic device according to claim 18, wherein: For pre-processing, the one or more processors are configured to extract features corresponding to the color filter array input image for each color type of the color filter array based on the pattern information.

22. The electronic device according to claim 21, wherein: For pre-processing, the one or more processors are configured to downsample the features.

23. The electronic device according to claim 18, wherein There is an artificial neural network model for each color type of the color filter array, and To generate the inferred image, the one or more processors are configured to generate the inferred image by inputting the pre-processed color filter array input image into an artificial neural network model corresponding to each color type of the color filter array.

24. The electronic device according to claim 18, wherein: The pattern information includes any one or both of: pixel position information of a color filter array associated with a pixel position and color information based on the pixel position.

25. The electronic device according to claim 24, wherein: The color information includes: any one or any combination of any two or more of color channel information, gain information and disparity information according to the pixel position.

26. The electronic device according to claim 18, wherein: The one or more processors are configured to post-process the output image based on the pattern information.

27. The electronic device according to claim 26, wherein: For post-processing, the one or more processors are configured to perform any one or both of denoising and super-resolution on the output image based on the pattern information.

28. The electronic device according to any one of claims 18 to 27, wherein: The color filter array input image includes any one of a Bayer pattern image, a four-pixel-in-one pattern image, and a nine-pixel-in-one pattern image, or any combination of any two or more of them.

29. The electronic device according to any one of claims 18 to 27, wherein: To generate the inferred image, the one or more processors are configured to generate red, green, and blue RGB images corresponding to the pre-processed color filter array input image.

30. An electronic device comprising: One or more processors configured to: obtaining a color filter array input image; obtaining pattern information corresponding to a color filter array; Preprocessing the color filter array input image based on the pattern information; generating an inferred image based on the preprocessed color filter array input image; as well as generating an output image by post-processing the inferred image based on the pattern information, The pattern information includes any one or both of pixel position information of a color filter array associated with a pixel position and color information based on the pixel position.

Images