Electronic device that performs video HDR processing based on multiple image data

Abstract

An electronic device performs HDR processing by using a plurality of image sensor blocks, the plurality of image sensor blocks including a first image sensor block for photographing an object with a first field of view, and a second image sensor block for photographing an object with a second field of view larger than the first field of view. An image signal processor generates first image data and second image data based on the photographed object. A main processor crops image data corresponding to the image data of the first sensor among the image data of the second sensor, and performs HDR processing based on the first image data and the cropped second image data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2019-0116781 filed on September 23, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] Embodiments of the inventive concepts described herein relate to electronic devices, and more particularly, to electronic devices for implementing video high dynamic range (HDR) processing based on multiple image sensors. Background Art

[0004] The image capture device is used to capture a photo of an image that a user wants to include an object that the user focuses on and a background. The image capture device includes various electronic circuits for collecting light and generating signals associated with the image, and provides the user with a service of capturing an image depending on the operation of the electronic circuits.

[0005] Image capture devices are popular and widely used by many users. Therefore, it is necessary to meet various requirements of users with respect to the performance and functions of the image capture devices.

[0006] For example, it may be desirable to improve the processing performance and operation speed of an image capture device to improve the user experience when operating the image capture device. For example, it may be necessary to appropriately control the method of operating the image capture device to reduce the power consumption of the image capture device. Thus, solutions for improving the structure and operation of the image capture device are being proposed to meet various requirements of users. Summary of the invention

[0007] Embodiments of the inventive concept provide an electronic device configured to perform video high dynamic range (HDR) processing based on image data obtained by a plurality of image sensors.

[0008] According to an embodiment, an electronic device is provided, including: a first image sensor block configured to capture an object with a first field of view and generate a first data signal corresponding to the object captured with the first field of view; a second image sensor block configured to capture an object with a second field of view larger than the first field of view and generate a second data signal corresponding to the object captured with the second field of view; an image signal processor configured to generate first image data of a first resolution based on the first data signal and second image data of a second resolution based on the second data signal; and a main processor configured to obtain cropped image data from among the second image data corresponding to the first image data, and perform high dynamic range (HDR) processing based on the first image data and the cropped image data.

[0009] According to an embodiment, an electronic device is provided, including: a first image sensor block configured to capture an object with a first field of view and generate a first data signal corresponding to the object captured with the first field of view; a second image sensor block configured to capture an object with a second field of view larger than the first field of view and generate a second data signal corresponding to the object captured with the second field of view; a third image sensor block configured to capture an object with a third field of view larger than the second field of view and generate a third data signal corresponding to the object captured with the third field of view; an image signal processor configured to generate first image data of a first resolution based on the first data signal, second image data of a second resolution based on the second data signal, and third image data of a third resolution based on the third data signal; and a main processor configured to perform high dynamic range (HDR) processing based on the first image data, the second image data, and the third image data, and in a first zoom factor mode, obtain cropped image data corresponding to the second image data from among the third image data, and perform HDR processing based on the second image data and the cropped image data.

[0010] According to an embodiment, a method for processing an image by using a first image sensor block having a first field of view and a second image sensor block having a second field of view larger than the first field of view is provided, the method comprising: photographing an object by the first image sensor block to generate a first data signal corresponding to the object photographed with the first field of view; photographing the object by the second image sensor block to generate a second data signal corresponding to the object photographed with the second field of view; generating first image data of a first resolution based on the first data signal; generating second image data of a second resolution based on the second data signal; generating cropped image data from among the second image data corresponding to the first image data; and performing high dynamic range (HDR) processing based on the first image data and the cropped image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects and features of the inventive concept will become apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings.

[0012] Figure 1 is a block diagram illustrating an exemplary configuration of an electronic device including an electronic circuit according to the embodiment.

[0013] Figure 2 It is a graphic Figure 1 FIG. 1 is a diagram of an exemplary configuration of an image processing block.

[0014] Figure 3 It is a graphic Figure 2 A block diagram of an exemplary operation of an image sensor block.

[0015] Figure 4 It is a graphic Figure 3A conceptual diagram of exemplary characteristics of an image sensor block.

[0016] Figure 5 It is a graphic Figure 2 Timing diagram of the operation of the image sensor block and the host processor.

[0017] Figure 6 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0018] Figure 7 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0019] Figure 8 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0020] Fig. 9 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0021] Fig.10 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0022] Fig.11 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0023] Fig.12 is a flowchart illustrating an image processing method according to an embodiment.

[0024] Fig.13 It is a graphic Figure 1 A block diagram of an exemplary configuration of an image processing block.

[0025] Fig.14 It is a graphic Fig.13 A block diagram of an exemplary operation of an image sensor block.

[0026] Fig.15 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0027] Fig.16 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0028] Fig.17 is a diagram illustrating the operation of an image sensor block according to a zoom factor.

[0029] Fig.18 is a diagram illustrating the operation of the image sensor block in the first zoom mode.

[0030] Fig.19 is a diagram illustrating the operation of the image sensor block in the second zoom mode.

[0031] Fig. 20 It is a graphic Fig.13 A block diagram of an exemplary operation of an image sensor block.

[0032] Fig.21 is a diagram illustrating the operation of an image sensor block according to a zoom factor.

[0033] Fig. 22 is a diagram illustrating the operation of an image sensor block according to a zoom factor.

[0034] Fig.23 The diagram depends on Fig.13 A conceptual diagram of an exemplary method for adjusting magnification based on a command received by a user interface. DETAILED DESCRIPTION

[0035] In the following, embodiments of the inventive concept may be described in detail and clearly to the extent that a person skilled in the art can easily implement the inventive concept. In the specification, "object", "background", "scene", etc. may refer to a scene to be referred to by a reference. Figures 1 to 18 An image of image data obtained by an electronic device is described.

[0036] Figure 1 is a block diagram illustrating an exemplary configuration of an electronic device including an electronic circuit according to the embodiment.

[0037] The electronic device 1000 may include various electronic circuits. For example, the electronic circuits of the electronic device 1000 may include an image processing block 1100, a communication block 1200, an audio processing block 1300, a buffer memory 1400, a nonvolatile memory 1500, a user interface 1600, a sensor 1700, a main processor 1800, and a power manager 1900.

[0038] The image processing block 1100 may include a lens 1110, an image sensor 1120, and an image signal processor 1150. For example, light may be reflected by an external object, background, and scene that the user intends to capture, and the lens 1110 may receive the reflected light. The image sensor 1120 may generate an electrical signal based on the light received through the lens 1110. The image signal processor 1150 may generate image data associated with an image of an external object, background, and scene by appropriately processing the electrical signal generated by the image sensor 1120. For ease of description, the term "object" may be used to indicate the entirety of a background, scene, etc., as well as an object that a photograph is aimed at or particularly focused on.

[0039] The image sensor 1120 may include pixels arranged in a configuration along rows and columns, and the image sensor 1120 may convert light incident on the pixels into electrical signals. The characteristics of the electrical signals (e.g., current intensity and voltage value) may vary with the characteristics of the received light (e.g., intensity). For example, the image sensor 1120 may be a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, etc.

[0040] Figure 1 1 and 11. In the figure, a lens 1110 and an image sensor 1120 are illustrated. However, in some embodiments, the image processing block 1100 may include multiple lenses and multiple image sensors. Multiple image sensors may be provided to have different functions, different performances, and / or different characteristics. For example, each of the multiple image sensors may include multiple lenses with different fields of view (FOV).

[0041] The image signal processor 1150 may include hardware circuits (e.g., analog circuits and logic circuits) configured to perform the operations to be described in the present disclosure. Additionally or alternatively, the image signal processor 1150 may include one or more processor cores and may execute program codes configured to provide the operations to be described in the present disclosure.

[0042] Examples are Figure 1 1, and the image signal processor 1150 is included in the image processing block 1100. However, in some embodiments, the image signal processor 1150 may be provided as part of the image sensor 1120, may be provided on a circuit or chip independent of the image processing block 1100, and / or may be provided as part of the main processor 1800. It is well understood that the present disclosure may be variously changed or modified with respect to the hardware and software configurations and arrangements of components herein, without limitation. Figure 1 .

[0043] The communication block 1200 may exchange signals with an external device / system via an antenna 1210. The transceiver 1220 and the modem (modulator / demodulator) 1230 of the communication block 1200 may process the exchanged signals according to various communication protocols. For example, the transceiver 1220 and the MODEM 1230 of the communication block 1200 may process the signals exchanged with the external device / system according to a wireless communication protocol such as Long Term Evolution (LTE), World Wide Interoperability for Microwave Access (WiMax), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Bluetooth, Near Field Communication (NFC), Wireless Fidelity (Wi-Fi), or Radio Frequency Identification (RFID).

[0044] The audio processing block 1300 may process sound information by using the audio signal processor 1310. The audio processing block 1300 may receive audio input through the microphone 1320 or may output audio through the speaker 1330.

[0045] The buffer memory 1400 may temporarily store data used for the operation of the electronic device 1000 (e.g., data processed by or to be processed by the main processor 1800). For example, the buffer memory 1400 may include a volatile / non-volatile memory such as a static random access memory (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a phase change RAM (PRAM), a magnetoresistive RAM (MRAM), a resistive RAM (ReRAM), or a ferroelectric RAM (FRAM). For example, an image captured by the image processing block 1100 or a video composed of images captured by the image processing block 1100 may be stored in the buffer memory 1400. The main processor 1800 may perform HDR processing on one or more images stored in the buffer memory 1400.

[0046] The non-volatile memory 1500 can store data regardless of whether power is supplied. For example, the non-volatile memory 1500 may include a non-volatile memory such as a flash memory, a PRAM, an MRAM, a ReRAM, and a FRAM. For example, the non-volatile memory 1500 may include a removable memory such as a hard disk drive (HDD), a solid state drive (SSD), a secure digital (SD) card, a universal flash storage device (UFS), and / or an embedded memory such as an embedded multimedia card (eMMC).

[0047] The user interface 1600 enables communication between the user and the electronic device 1000 so that the user can control the electronic device 1000 via command input through the user interface 1600. For example, the user interface 1600 may include input interfaces such as a keypad, buttons, a touch screen, a touch pad, a visual sensor, a motion sensor, and a gyro sensor. The user interface 1600 may also include output interfaces such as a liquid crystal display (LCD) device, a light emitting diode (LED) display device, an organic LED (OLED) display device, an active matrix OLED (AMOLED) display device, a motor, and an LED lamp for providing information about the operation of the electronic device 1000 to the user.

[0048] The sensor 1700 may sense various types of physical energy provided from the outside of the electronic device 1000. For example, the sensor 1700 may sense a transmission medium of physical energy, such as temperature, voice, and light. For example, the sensor 1700 may sense illuminance, and may transmit data indicating the sensed illuminance to the main processor 1800.

[0049] The main processor 1800 may perform various operations for controlling the overall operation of the electronic device 1000. For example, the main processor 1800 may be implemented as a general-purpose processor, a dedicated processor, or an application processor and may include one or more processor cores. The main processor 1800 may control the image processing block 1100 for obtaining image data associated with an object outside the electronic device 1000. For example, the main processor 1800 may control the image processing block 1100 based on information provided from the sensor 1700.

[0050] The power manager 1900 may appropriately convert power received from a battery or an external power source. The power manager 1900 may supply the converted power to the components of the electronic device 1000.

[0051] Figure 1 The exemplary components shown in FIG. 1 are provided for better understanding and are not intended to limit the present disclosure. The electronic device 1000 may not include Figure 1 One or more components shown in, or may further include Figure 1 At least one component not shown.

[0052] Figure 2 It is a graphic Figure 1 FIG. 1 is a diagram of an exemplary configuration of an image processing block.

[0053] The image processing block 1100 may include first and second image sensor blocks 1101 and 1102 and an image signal processor 1150. The user interface 1600, the illumination sensor 1710, and the main processor 1800 are illustrated together to more fully describe the operation of the image processing block 1100. For example, Figure 1 A portion of the sensor 1700 implements an illumination sensor 1710 .

[0054] The image processing block 1100 may be implemented by a dual sensor structure including first and second image sensors 1121 and 1122. For example, the first image sensor block 1101 may include a first lens 1111, a first actuator 1141, and a first image sensor 1121. Similarly, the second image sensor block 1102 may include a second lens 1112, a second actuator 1142, and a second image sensor 1122.

[0055] For example, the object 10 may be a target that a user of the electronic device 1000 intends to photograph. In response to a user's request, the image processing block 1100 may generate signals and data associated with an image of the object 10 under the control of the main processor 1800. Light may be reflected from the object 10, and the reflected light may be received by the first and second lenses 1111 and 1112. For example, light collected by the first lens 1111 may be provided to the first image sensor 1121, and light collected by the second lens 1112 may be provided to the second image sensor 1122.

[0056] The first and second lenses 1111 and 1112 may be configured to collect light from an object at which the camera is directed with various fields of view. For example, the field of view of light received by the first lens 1111 may be different from the field of view of light received by the second lens 1112. The first lens 1111 may receive light with a narrower field of view than the field of view of the second lens 1112. Conversely, the second lens 1112 may receive light with a wider field of view than the field of view of the first lens 1111.

[0057] The first image sensor 1121 may generate a first signal D1 based on the light received from the first lens 1111, and the second image sensor 1122 may generate a second signal D2 based on the light received from the second lens 1112. For example, a first image based on the first signal D1 may be associated with an image including a relatively narrow field of view of the object 10, and the second signal D2 may be associated with an image including a relatively wide field of view of the object 10.

[0058] The first and second image sensors 1121 and 1122 may be provided to have different functions, different performances, and / or different characteristics. For example, the first image sensor 1121 may include a relatively low-resolution pixel array to process light received by the first lens 1111 having a relatively narrow field of view. Conversely, the second image sensor 1122 may include a relatively high-resolution pixel array to process light received by the second lens 1112 having a relatively wide field of view. Alternatively, the resolution of the pixel array of the first image sensor 1121 and the resolution of the pixel array of the second image sensor 1122 may be equal.

[0059] As a result, the dynamic range of the first image based on the first signal D1 output from the first image sensor 1121 may be different from the dynamic range of the second image based on the second signal D2 output from the second image sensor 1122. For example, the first image based on the first signal D1 may include a large number of relatively dark areas, and the second image based on the second signal D2 may include a large number of relatively bright areas. However, the properties of the dynamic ranges of the first image and the second image are not limited thereto, and the properties of the dynamic ranges may depend on various factors of the first and second image sensors 1121 and 1122, such as illumination and exposure time.

[0060] In some embodiments, the image processing block 1100 may further include first and second actuators 1141 and 1142 for adjusting the focal lengths of the first and second lenses 1111 and 1112. The first actuator 1141 may move the position of the first lens 1111, and thus, may adjust the focal length of the first lens 1111. The second actuator 1142 may move the position of the second lens 1112, and thus, may adjust the focal length of the second lens 1112. For example, each of the first and second actuators 1141 and 1142 may be a moving actuator.

[0061] For example, the focal lengths of the first and second lenses 1111 and 1112 may be adjusted by the first and second actuators 1141 and 1142 so that the focal lengths of the first and second lenses 1111 and 1112 are different. For example, the first and second actuators 1141 and 1142 may include physical devices such as motors for moving the first and second lenses 1111 and 1112, respectively.

[0062] In addition, the image signal processor 1150 can perform various processing on the signals D1 and D2. For example, the image signal processor 1150 can perform various signal processing such as pixel correction, demosaicing, noise reduction, lens shading correction, gamma correction, and edge enhancement. The signal processed by the image signal processor 1150 can be output to the main processor 1800.

[0063] The main processor 1800 may perform high dynamic range (HDR) processing based on the signal received from the image signal processor 1150. Thus, final image data may be generated. Exemplary HDR processing performed by the main processor 1800 will be described more fully with respect to subsequent embodiments.

[0064] The HDR-processed image data may be stored in the internal memory of the main processor 1800, the buffer memory 1400, and / or the nonvolatile memory 1500. The HDR-processed image data may indicate attributes such as shape, color, and motion of the object 10. The HDR-processed image data may be provided to a user through an output interface (e.g., a display device) of the user interface 1600.

[0065] The main processor 1800 may control the operations of the first and second image sensors 1121 and 1122, the image signal processor 1150, and the first and second actuators 1141 and 1142. For example, the main processor 1800 may send signals to various components to control the operation timing of the first and second image sensors 1121 and 1122, the image signal processor 1150, and the first and second actuators 1141 and 1142 in response to a user request (e.g., a button press, a screen touch, a motion, or a gesture) recognized through the user interface 1600.

[0066] For example, the illuminance of the photographing environment may be sensed by the illuminance sensor 1710. Data associated with the illuminance may be provided from the illuminance sensor 1710 to the main processor 1800, and the main processor 1800 may control the first and second image sensor blocks 1101 and 1102 by using control signals FB1 and FB2 based on the provided data.

[0067] For example, the user may input a "zoom in" or "zoom out" command via an input interface of the user interface 1600 for capturing an image. The zoom in command may be associated with capturing an object with a narrow field of view, and the zoom out command may be associated with capturing an object with a wide field of view.

[0068] For example, the zoom in or zoom out command may include information of a higher or lower zoom ratio. The zoom ratio may be a value indicating the depth of the object to be photographed from the electronic device, or the width of the area including the object to be photographed. The main processor 1800 may control the first and second actuators 1141 and 1142 based on the information indicated by the zoom ratio.

[0069] For example, the user can set the zoom condition for capturing the desired image by slowly or gradually increasing or decreasing the value of the zoom factor (e.g., stepwise or gradually) or selecting a specific value of the zoom factor. To this end, the input interface of the user interface 1600 can receive the zoom factor from the user. HDR processing based on this user zoom factor will be described more fully with respect to subsequent embodiments.

[0070] The main processor 1800 may execute instruction sets of various program codes. For example, the main processor 1800 may execute instruction sets of an application (AP), a camera hardware abstraction layer (HAL), a camera driver (DRV), an image processing library (LIB), etc. The application may be executed in response to a user's request, and the application may be, for example, a camera application, a movie making application, etc.

[0071] Figure 3 It is a graphic Figure 2 A block diagram of an exemplary operation of an image sensor block.

[0072] refer to Figure 2 and Figure 3 , the first image sensor block 1101 may include a first image sensor 1121 for generating an image captured with a first field of view FOV1 and a second image sensor 1122 for generating an image captured with a second field of view FOV2. In addition, in this specification, "an image or image data having a specific field of view (or a specific field of view)" may refer to "image data or an image output from an image sensor including a lens having a specific field of view".

[0073] In an embodiment, the second field of view FOV2 may be larger than the first field of view FOV1. Therefore, the first image sensor block 1101 may be operated to sense an image of a relatively narrow field of view, and the second image sensor block 1102 may be operated to sense an image of a relatively wide field of view. For example, the first lens 1111 may be a wide-angle lens, and the second lens 1112 may be an ultra-wide-angle lens. Alternatively, the first lens 1111 may be a telephoto lens, and the second lens 1112 may be a wide-angle lens.

[0074] For example, the first image sensor block 1101 may receive light reflected by an object in a region R1 corresponding to the first field of view FOV1, and may generate a first signal D1 based on the received light. The second image sensor block 1102 may receive light reflected by an object in a region R2 corresponding to the second field of view FOV2, and may generate a second signal D2 based on the received light.

[0075] The image signal processor 1150 may generate first image data IDAT1 and second image data IDAT2 based on the first signal D1 and the second signal D2, respectively. The dynamic range of the first image data IDAT1 and the dynamic range of the second image data IDAT2 may be different from each other. For example, the first image data IDAT1 may include a relatively dark brightness value, and the second image data IDAT2 may include a relatively bright brightness value.

[0076] Then, the main processor 1800 may perform HDR processing based on the first image data IDAT1 and the second image data IDAT2 having different brightness values. As a result, since an image having a wider dynamic range is obtained through HDR processing, the quality of an image or a video composed of the image may be improved.

[0077] For example, the first and second image sensor blocks 1101 and 1102 may adjust the amount of light received from the external surroundings of the electronic device 1000 under the control of the main processor 1800. In an embodiment, the main processor 1800 may adjust the shutter speeds of the first and second image sensors 1121 and 1122 to adjust the amount of light collected from the surroundings of the electronic device 1000. In another embodiment, the main processor 1800 may adjust the aperture values ​​of the first and second image sensors 1121 and 1122 to adjust the amount of light collected from the surroundings of the electronic device 1000.

[0078] To adjust the shutter speed and / or aperture value of the first and second image sensors 1121 and 1122, the main processor 1800 may send a signal to control a mechanical device included in the first and second image sensors 1121 and 1122, or may send a signal to control pixels included in the first and second image sensors 1121 and 1122. For example, the main processor 1800 may control the operation of transistors included in the pixels of the first and second image sensors 1121 and 1122.

[0079] Figure 4 It is a graphic Figure 3 A conceptual diagram of exemplary characteristics of an image sensor block. Figure 2 and Figure 4 The operation of the first and second image sensor blocks 1101 and 1102 having different operating characteristics is described. Figure 4 In the example, the x-axis represents the characteristic value.

[0080] The characteristic value may be associated with a brightness value of image data based on a signal output from the image sensor. As an example of a characteristic value, a brightness range of first image data based on a first signal D1 output from the first image sensor block 1101 may be different from a brightness range of second image data based on a second signal D2 output from the second image sensor block 1102. For example, a maximum value (i.e., "c") of the brightness range of the first image data may be smaller than a maximum value (i.e., value "d") of the brightness range of the second image data. For example, at least a portion of the brightness range of the first image data may overlap with at least a portion of the brightness range of the second image data, or may not overlap with at least a portion of the brightness range of the second image data.

[0081] As another example of a characteristic value, a dynamic range of first image data based on a first signal D1 output from the first image sensor block 1101 may be different from a dynamic range of second image data based on a second signal D2 output from the second image sensor block 1102. For example, a maximum value of the dynamic range of the first image data (i.e., a value of "c") may be smaller than a maximum value of the dynamic range of the second image data (i.e., a value of "d"). A relatively large value of the dynamic range may mean that a ratio of a bright area in an image is correspondingly large. For example, at least a portion of the dynamic range of the first image data may overlap with at least a portion of the dynamic range of the second image data, or may not overlap with at least a portion of the dynamic range of the second image data.

[0082] As another example of a characteristic value, the characteristic value may be a value associated with a physical characteristic of the image sensor itself. For example, the characteristic value may be associated with the number of physical pixels constituting the pixel array. In this configuration, the characteristic value may be represented by a range. Figure 4 Unlike the example of , the characteristic value may be a single fixed value. For example, the resolution of the pixel array of the first image sensor block 1101 and the resolution of the pixel array of the second image sensor block 1102 may be different from each other. For example, the first image sensor block 1101 may include a pixel array adapted to process light received by a first lens 1111 having a relatively narrow field of view. The second image sensor block 1102 may include a pixel array adapted to process light received by a second lens 1112 having a relatively wide field of view.

[0083] Figure 5 It is a graphic Figure 2 Timing diagram of the operation of the image sensor block and the host processor.

[0084] Between the first time t1 and the third time t3, the first and second image sensor blocks 1101 and 1102 may operate to obtain image data corresponding to a current image frame. The first and second image sensor blocks 1101 and 1102 may output signals D1 and D2 indicating the obtained image data, respectively.

[0085] After the first time t1, the image signal processor 1150 may receive signals D1 and D2 from the first and second image sensor blocks 1101 and 1102. The image signal processor 1150 may obtain first image data and second image data based on the signals D1 and D2. Between the second time t2 and the fourth time t4, the main processor 1800 may perform HDR processing based on the first image data and the second image data corresponding to different fields of view, respectively.

[0086] exist Figure 5In the example of , the first and second image sensor blocks 1101 and 1102 can operate during the same time period to obtain image data of different brightness values ​​to be used for HDR processing. When the first and second image sensor blocks 1101 and 1102 operate during the same time period, image data to be used for HDR processing can be obtained faster in real time. Therefore, the electronic device 1000 can obtain image data to be HDR-processed by the operation of the first and second image sensor blocks 1101 and 1102 and the image signal processor 1150 in real time.

[0087] Figure 6 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0088] Reference Figure 2 , Figure 4 and Figure 6 Describe how to perform HDR processing for two images with different fields of view. The image signal processor 1150 can obtain first image data IDAT1 based on the first signal D1 received from the first image sensor block 1101, and can obtain second image data IDAT2 based on the second signal D2 received from the second image sensor block 1102. For example, the first and second image sensor blocks 1101 and 1102 can respectively obtain the first image data IDAT1 and the second image data IDAT2 by sensing light received from lenses with different fields of view.

[0089] For example, the first image sensor 1121 may include a pixel array consisting of "N" columns each including "M" pixels (or consisting of "M" rows each including "N" pixels). Therefore, the size of the first image data IDAT1 may be "M pixels×N pixels". Similarly, the second image sensor 1122 may include a pixel array consisting of "P" columns each including "Q" pixels (or consisting of "Q" rows each including "P" pixels). Therefore, the size of the second image data IDAT2 may be "P pixels×Q pixels". In an embodiment, "P" may be greater than "M", and "Q" may be greater than "N". That is, the resolution of the second image data IDAT2 may be greater than the resolution of the first image data IDAT1. Therefore, the brightness value (or range of brightness values) of the first image data IDAT1 and the brightness value (or range of brightness values) of the second image data IDAT2 may be different from each other.

[0090] However, in another embodiment, "P" and "M" may be equal, and "Q" and "N" may be equal. In this configuration, the resolution of the first image data IDAT1 may be equal to the resolution of the second image data IDAT2, and the brightness value (or the range of brightness values) of the first image data IDAT1 and the brightness value (or the range of brightness values) of the second image data IDAT2 may be different from each other. However, for the convenience of description in the specification, the description will be given under the assumption that the resolutions of the image data respectively output from the image sensor blocks are different. In addition, as an assumption for obtaining HDR processing with a wide dynamic range, the embodiment is described on the assumption that the brightness values ​​(or the range of brightness values) of the image data respectively output from the image sensor blocks are different.

[0091] Hereinafter, for easy understanding, "M pixels×N pixels" as the size of image data is referred to as "M×N." In the specification, the case where the size of image data is represented by "M×N" means that the image data indicates data represented by discontinuous units (e.g., pixels) included in M ​​rows and N columns.

[0092] As described above, the second image sensor 1122 may include a pixel array consisting of "Q" columns each including "P" pixels (or consisting of "P" rows each including "Q" pixels). That is, the size of the second image data IDAT2 may be "P×Q".

[0093] The main processor 1800 may crop a portion of the second image data IDAT2. The main processor 1800 may crop a portion of the second image data IDAT2 corresponding to the first image data IDAT1. The size of the image data of the second image data IDAT2 belonging to the area corresponding to the first image data IDAT1 may be "M×N". For example, the size "M×N" of the cropped area may be, but is not limited to, half or less of the size "P×Q" of the second image data IDAT2. As a result of the cropping, the third image data IDAT3 of "M×N" may be generated.

[0094] As reference Figure 4As described, the first and second image sensor blocks 1101 and 1102 can respectively obtain the first image data IDAT1 and the second image data IDAT2 having different brightness values ​​through the characteristics of the image sensor blocks themselves. As a result, the brightness value (or the range of brightness values) of the first image data IDAT1 and the brightness value (or the range of brightness values) of the third image data IDAT3 can be different from each other. The main processor 1800 can receive the first image data IDAT1 and the third image data IDAT3 from the image signal processor 1150, and can perform HDR processing based on the received image data. The main processor 1800 can obtain the image data IOUT corresponding to the current video frame through HDR processing.

[0095] The image indicated by the HDR-processed image data IOUT may include the dynamic range of the first image data IDAT1 and the dynamic range of the third image data IDAT3. In other words, an image having a wider dynamic range (i.e., "a" to "d") than the dynamic range of the first image data IDAT1 (i.e., "a" to "c") or the dynamic range of the third image data IDAT3 (i.e., "b" to "d") may be obtained through HDR processing.

[0096] at the same time, Figure 6 The embodiments may relate to HDR processing for video. However, when the HDR processing method of the present disclosure is applied to photos, the brightness value of the image data may be manually adjusted by adjusting the exposure value. For example, when the surrounding brightness is very bright, the illumination sensor 1710 (refer to Figure 2 ) can sense the surrounding brightness, and can transmit information about the surrounding brightness to the main processor 1800. The main processor 1800 can adjust the exposure value of at least one of the first and second image sensor blocks 1101 and 1102 based on the surrounding brightness information. As a result, an image with a wider dynamic range can be obtained through HDR processing.

[0097] Figure 7 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0098] Reference Figure 2 and Figure 7Describes how to perform HDR processing for two images with different fields of view. The image signal processor 1150 may obtain first image data IDAT1 based on a first signal D1 received from the first image sensor block 1101, and may obtain second image data IDAT2 based on a second signal D2 received from the second image sensor block 1102. The size of the first image data IDAT1 may be M'×N', and the size of the second image data IDAT2 may be "P×Q". In an embodiment, "P" may be greater than M', and "Q" may be greater than N'. That is, the resolution of the second image data IDAT2 may be greater than the resolution of the first image data IDAT1. For example, the brightness value (or range of brightness values) of the first image data IDAT1 and the brightness value (or range of brightness values) of the second image data IDAT2 may be different from each other.

[0099] In an embodiment, the resolution of the first image data IDAT1 corresponding to M'×N' may be greater than the resolution of the region of the second image data IDAT2 corresponding to the first image data IDAT1 (i.e., M×N). That is, "M" may be smaller than M', and "N" may be smaller than N'. In this configuration, before performing HDR processing, the resolution of the image data that is the basis of HDR processing does not need to be adjusted.

[0100] For example, the main processor 1800 may perform binning on the first image data IDAT1. The main processor 1800 may perform binning depending on various types of algorithms. The main processor 1800 may obtain "M×N" third image data IDAT3 by performing binning on the M'×N' first image data IDAT1.

[0101] The main processor 1800 may crop a portion of the second image data IDAT2. The main processor 1800 may crop image data corresponding to the third image data IDAT3 of the second image data IDAT2. Figure 7 The cropping described is similar to that of reference Figure 6 The cropping described above and thus further description will be omitted to avoid redundancy. The fourth image data IDAT4 of "M×N" may be generated as a result of the cropping.

[0102] Since the third image data IDAT3 is obtained by merging the area corresponding to the first image data IDAT1 and the fourth image data IDAT4 is obtained by performing cropping on the second image data IDAT2, the image indicated by the third image data IDAT3 and the image indicated by the fourth image data IDAT4 may correspond to each other. For example, the image indicated by the third image data IDAT3 and the image indicated by the fourth image data IDAT4 may be similar in most aspects except for the brightness value.

[0103] Because the luminance value (or range of luminance values) of the first image data IDAT1 and the luminance value (or range of luminance values) of the second image data IDAT2 are different from each other, the luminance value (or range of luminance values) of the third image data IDAT3 and the luminance value (or range of luminance values) of the fourth image data IDAT4 can be different from each other. The main processor 1800 can perform HDR processing based on the third image data IDAT3 and the fourth image data IDAT4 having different luminance values. The main processor 1800 can obtain the image data IOUT through the HDR processing.

[0104] According to Figure 7 the embodiment, since merging is performed on the first image data IDAT1 before the HDR processing, the amount of data to be subjected to the HDR processing is reduced. Accordingly, the speed of performing the HDR processing can be increased, and the power consumption of the main processor 1800 can be reduced accordingly.

[0105] Figure 8 FIG. is a diagram illustrating operations for HDR processing according to an embodiment.

[0106] In Figure 7 the embodiment, an HDR processing associated with the following configuration is described: wherein the resolution of the first image data IDAT1 is greater than the resolution of the region corresponding to the first image data IDAT1 of the second image data IDAT2 (i.e., M×N). Conversely, Figure 8 the embodiment of involves an HDR processing associated with the following configuration: the resolution of the first image data IDAT1 (i.e., M×N) is less than the resolution of the region corresponding to the first image data IDAT1 of the second image data IDAT2 (i.e., M’×N’) (i.e., M<M’ and N<N’).

[0107] The image signal processor 1150 can generate the first image data IDAT1 of “M×N” based on the first signal D1 output from the first image sensor block 1101. The image signal processor 1150 can generate the second image data IDAT2 of “P×Q” based on the second signal D2 output from the second image sensor block 1102.

[0108] The main processor 1800 may crop the image data of the second image data IDAT2 corresponding to the first image data IDAT1. As a result of the cropping, the third image data IDAT3 of M'×N' may be generated. However, since the resolution of the first image data IDAT1 and the resolution of the third image data IDAT3 are different from each other, the HDR process should be performed immediately with a delay. Therefore, an additional operation on the third image data IDAT3 may be required. For example, the main processor 1800 may obtain the fourth image data IDAT4 of "M×N" by performing merging on the third image data IDAT3 of M'×N'. As a result of the cropping and merging, the size of the first image data IDAT1 and the size of the fourth image data IDAT4 may be equal, that is, may be "M×N".

[0109] Because the brightness value (or range of brightness values) of the first image data IDAT1 and the brightness value (or range of brightness values) of the second image data IDAT2 are different from each other, the brightness value (or range of brightness values) of the first image data IDAT1 and the brightness value (or range of brightness values) of the fourth image data IDAT4 may be different from each other. The main processor 1800 may perform HDR processing based on the first image data IDAT1 and the fourth image data IDAT4 having different brightness values. The main processor 1800 may obtain the image data IOUT through HDR processing.

[0110] Fig. 9 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0111] exist Figure 8 In the embodiment of FIG. 1 , an additional operation is described for the second image data IDAT2 when the resolution of the first image data IDAT1 is smaller than the resolution of the region of the second image data IDAT2 corresponding to the first image data IDAT1. Fig. 9 In the embodiment, when the resolution of the first image data IDAT1 is smaller than the resolution of the region of the second image data IDAT2 corresponding to the first image data IDAT1 (ie, M>M' and N>N'), an additional operation is performed on the first image data IDAT1.

[0112] The image signal processor 1150 may generate first image data IDAT1 of M′×N′ based on the first signal D1 output from the first image sensor block 1101 , and may generate second image data IDAT2 of “P×Q” based on the second signal D2 output from the second image sensor block 1102 .

[0113] The main processor 1800 may obtain the third image data IDAT3 of "M×N" by performing upscaling on the first image data IDAT1 of M'×N'. The main processor 1800 may perform upscaling depending on various types of algorithms. The main processor 1800 may crop a portion of the second image data IDAT2. The main processor 1800 may crop the image data of the second image data IDAT2 corresponding to the third image data IDAT3. As a result of the cropping, the fourth image data of "M×N" may be generated, and the size of the third image data IDAT3 and the size of the fourth image data IDAT4 may be equal.

[0114] Because the brightness value of the first image data IDAT1 and the brightness value of the second image data IDAT2 are different from each other, the brightness value of the third image data IDAT3 and the brightness value of the fourth image data IDAT4 may be different from each other. The main processor 1800 may perform HDR processing based on the third image data IDAT3 and the fourth image data IDAT4 having different brightness values. The main processor 1800 may obtain the image data IOUT through HDR processing.

[0115] Fig.10 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0116] Fig.10 An embodiment similar to Figure 7 However, in Fig.10 In the embodiment of FIG. 5 , no cropping may be performed on the second image data IDAT2 before HDR processing. Figures 7 to 9 In the embodiment of the present invention, the result of the HDR processing may be provided to the user through a user interface (e.g., a display) depending on the size of the image captured by the first image sensor block 1101 (or the size of the image obtained as a result of performing merging on the image captured by the first image sensor block 1101). Fig.10 In the embodiment, the result of HDR processing may be provided to the user through the user interface depending on the size of the image captured by the second image sensor block 1102.

[0117] The image signal processor 1150 may obtain the first image data IDAT1 of M'×N' based on the first signal D1 received from the first image sensor block 1101, and may obtain the second image data IDAT2 of "P×Q" based on the second signal D2 received from the second image sensor block 1102. However, since the size of the first image data IDAT1 and the size of the area of ​​the second image data IDAT2 corresponding to the first image data IDAT1 are different from each other, additional processing is required with respect to the first image data IDAT1.

[0118] For example, the main processor 1800 may obtain the third image data IDAT3 of "M×N" by performing merging on the first image data IDAT1 of M'×N. The main processor 1800 may perform HDR processing based on the third image data IDAT3 and the second image data IDAT2, and may obtain the image data IOUT through the HDR processing.

[0119] Fig.11 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0120] refer to Figures 6 to 9 The described embodiments may be associated with HDR processing of each frame of a video captured in real time or HDR processing of an image captured at a certain moment. Fig.11 Embodiments of may be associated with HDR processing of an image captured at a certain moment. Alternatively, Fig.11 Embodiments may be associated with HDR processing of multiple images extracted from a pre-captured video.

[0121] In a configuration in which a time period between when the first image data IDAT1 is obtained by the first image sensor block 1101 and when the third image data IDAT3 is obtained is sufficiently short, an image indicated by the first image data IDAT1 and an image indicated by the third image data IDAT3 may correspond to each other. As in the above description, an image indicated by the second image data IDAT2 and an image indicated by the fourth image data IDAT4 may correspond to each other.

[0122] The image signal processor 1150 may sequentially obtain image data IDAT1 and IDAT3 based on signals D1 and D3 sequentially received from the first image sensor block 1101. The image signal processor 1150 may sequentially obtain image data IDAT2 and IDAT4 based on signals D2 and D4 sequentially received from the second image sensor block 1102.

[0123] The size of the first image data IDAT1 and the size of the third image data IDAT3 may be M'×N', and the size of the second image data IDAT2 and the size of the fourth image data IDAT4 may be “P×Q.” For example, brightness values ​​of the first to fourth image data IDAT1 to IDAT4 may be different from each other.

[0124] The main processor 1800 may perform a merger on the first image data IDAT1 and the third image data IDAT3. The main processor 1800 may obtain the fifth image data IDAT5 of "M×N" by performing a merger on the first image data IDAT1 of M'×N', and may obtain the seventh image data IDAT7 of "M×N" by performing a merger on the third image data IDAT3 of M'×N'. The main processor 1800 may obtain the sixth image data IDAT6 of "M×N" by cropping the image data of "M×N" corresponding to the fifth image data IDAT5 or the seventh image data IDAT7 of the second image data IDAT2. The main processor 1800 may obtain the eighth image data IDAT8 of "M×N" by cropping the image data of "M×N" corresponding to the fifth image data IDAT5 or the seventh image data IDAT7 of the fourth image data IDAT4.

[0125] Since the brightness values ​​of the first to fourth image data IDAT1 to IDAT4 are different from each other, the brightness values ​​of the fifth to eighth image data IDAT5 to IDAT8 may be different from each other. The main processor 1800 may perform HDR processing based on the fifth to eighth image data IDAT5 to IDAT8 having different brightness values. The main processor 1800 may obtain image data IOUT through HDR processing.

[0126] However, the HDR processing method of the main processor 1800 is not limited thereto. For example, the main processor 1800 may perform HDR processing based on some (e.g., IDAT5, IDAT7, and IDAT8) of the fifth image data IDAT5 to the eighth image data IDAT8. For example, the main processor 1800 may perform a first HDR processing for the fifth image data IDAT5 and the sixth image data IDAT6, and may perform a second HDR processing for the seventh image data IDAT7 and the eighth image data IDAT8. The main processor 1800 may perform a third HDR processing based on the result of the first HDR processing and the result of the second HDR processing. In an embodiment, HDR processing may be performed based on various image data that can be combined from the image data IDAT5, IDAT6, IDAT7, and IDAT8.

[0127] at the same time, Fig.11 The embodiment described is based on reference Figure 7 The cropping and merging described above perform HDR processing, but the embodiment is not limited thereto. Figure 6 and Figures 8 to 10 At least one of the described HDR processing schemes may be applied to Fig.11 Embodiment of the invention.

[0128] Fig.12is a flowchart illustrating an image processing method according to an embodiment.

[0129] Combined with reference Figure 2 , Figure 3 and Fig.12 In operation S110, the first image sensor block 1101 generates a first signal D1 corresponding to the first field of view FOV1. For example, the first signal D1 may be based on light incident through the first lens 1111 having a relatively narrow first field of view FOV1. The image signal processor 1150 may generate first image data IDAT1 based on the first signal D1.

[0130] In operation S120, the second image sensor block 1102 generates a second signal D2 corresponding to the second field of view FOV2. The image signal processor 1150 may generate second image data IDAT2. For example, the second signal D2 may be based on light incident through the second lens 1112 having a relatively wide second field of view FOV2. The image signal processor 1150 may generate the second image data IDAT2 based on the second signal D2. For example, the range of brightness values ​​of the second image data IDAT2 may be different from the range of brightness values ​​of the first image data IDAT1.

[0131] In operation S130, it is determined whether the resolution of the first image data IDAT1 is equal to the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1. The fact that the resolution of the first image data IDAT1 is equal to the resolution of the portion corresponding to the first image data IDAT1 (S130-Yes) means that no additional processing for adjusting the resolution needs to be performed. Therefore, the process proceeds to operation S140.

[0132] In operation S140 , cropping is performed on a portion of the second image data IDAT2 corresponding to the first image data IDAT1 .

[0133] In operation S150, HDR processing is performed on the cropped portion of the first image data IDAT1 and the second image data IDAT2. Since the dynamic range of the first image data IDAT1 and the dynamic range of the second image data IDAT2 are different from each other, a wide dynamic range can be obtained through the HDR processing.

[0134] Meanwhile, returning to operation S130, when the resolution of the first image data IDAT1 is not equal to the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1 (S130-No), an additional process for adjusting the resolution is required before HDR processing. Therefore, the process proceeds to operation S160.

[0135] In operation S160, the main processor 1800 adjusts the resolution of the first image data IDAT1. For example, when the resolution of the first image data IDAT1 is greater than the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1, the main processor 1800 may perform merging on the first image data IDAT1. Conversely, when the resolution of the first image data IDAT1 is less than the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1, the main processor 1800 may perform resolution enhancement on the first image data IDAT1. When the resolution is adjusted by the main processor 1800, the resolution of the first image data IDAT1 may be equal to the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1.

[0136] In operation S170 , cropping is performed on a portion of the second image data IDAT2 corresponding to the resized first image data IDAT1 .

[0137] In operation S180, HDR processing is performed on the cropped portion of the resized first and second image data IDAT1 and IDAT2. Since the dynamic ranges of the first and second image data IDAT1 and IDAT2 are different from each other, a wide dynamic range can be obtained through HDR processing.

[0138] However, the inventive concept is not limited to the above-described embodiments. When the resolution of the first image data IDAT1 is different from the resolution of the portion of the second image data IDAT2 corresponding to the first image data IDAT1, Figure 7 , Figure 8 and Fig. 9 The described embodiments may be applied to Fig.13 Embodiment of the invention.

[0139] Fig.13 It is a graphic Figure 1 A block diagram of an exemplary configuration of an image processing block.

[0140] The image processing block 1100 may include first, second to nth image sensor blocks 1101, 1102 to 110n and an image signal processor 1150. To conceptually describe the image processing block 1100, a user interface 1600, an illuminance sensor 1710, and a main processor 1800 are illustrated together. Figure 1 A portion of the sensor 1700 implements an illumination sensor 1710 .

[0141] The first image sensor block 1101 may include a first lens 1111, a first actuator 1141, and a first image sensor 1121. The second image sensor block 1102 may include a second lens 1112, a second actuator 1142, and a second image sensor 1122. The nth image sensor block 110n may include an nth lens 111n, an nth actuator 114n, and an nth image sensor 112n (n is an integer of 3 or more). Fig.13 The configuration and operation of the first to nth image sensor blocks 1101 to 110n are similar to those of Figure 2 The configuration and operation of each of the first and second image sensor blocks 1101 and 1102 are described below, and therefore, additional description will be omitted to avoid redundancy.

[0142] Fig.14 It is a graphic Fig.13 A block diagram of an exemplary operation of an image sensor block.

[0143] In an embodiment, the image processing block 1100 may include first to third image sensor blocks 1101, 1102, and 1103. The first image sensor block 1101 may include a first image sensor 1121 for generating an image of a region R1 corresponding to a first field of view FOV1, a second image sensor 1122 for generating an image of a region R2 corresponding to a second field of view FOV2, and a third image sensor 1123 for generating an image of a region R3 corresponding to a third field of view FOV3.

[0144] The third field of view FOV3 may be larger than the second field of view FOV2, and the second field of view FOV2 may be larger than the first field of view FOV1. For example, the first lens 1111 may be a telephoto lens, the second lens 1112 may be a wide-angle lens, and the third lens 1113 may be an ultra-wide-angle lens. For example, the field of view of the first lens 1111 may be 10 to 15 degrees, the field of view of the second lens 1112 may be 70 to 90 degrees, and the field of view of the third lens 1113 may be 100 to 130 degrees. However, the fields of view of the first, second, and third lenses 1111, 1112, and 1113 are not limited thereto.

[0145] For example, the first image sensor block 1101 may include a relatively low-resolution pixel array, the third image sensor block 1103 may include a relatively high-resolution pixel array, and the second image sensor block 1102 may include a pixel array with a resolution between the relatively low resolution and the relatively high resolution. For example, the first image sensor block 1101 may be operated to sense an image of a region R1 having a relatively narrow field of view, the third image sensor block 1103 may be operated to sense an image of a region R3 having a relatively wide field of view, and the second image sensor block 1102 may be operated to sense an image of a region R2 having an intermediate field of view.

[0146] For example, like the above-described embodiment, the resolutions of the pixel arrays of at least two of the first, second, and third image sensor blocks 1101, 1102, and 1103 may be equal, or the resolutions of the pixel arrays of the first, second, and third image sensor blocks 1101, 1102, and 1103 may be different from each other. However, for convenience, a description is given under the assumption that the resolutions of the pixel arrays of the first, second, and third image sensor blocks 1101, 1102, and 1103 are different from each other. In addition, as an assumption for obtaining HDR processing with a wide dynamic range, there may be a corresponding assumption that the brightness values ​​(or ranges of brightness values) of the image data respectively output from the image sensor blocks are different from each other.

[0147] The image signal processor 1150 may obtain first image data IDAT1 based on the first signal D1 received from the first image sensor block 1101, may obtain second image data IDAT2 based on the second signal D2 received from the second image sensor block 1102, and may obtain third image data IDAT3 based on the third signal D3 received from the third image sensor block 1103. The first, second, and third image sensor blocks 1101, 1102, and 1103 may obtain image data IDAT1, IDAT2, and IDAT3, respectively.

[0148] Meanwhile, in addition to the image processing block including three image sensor blocks, Fig.14 An embodiment similar to Figure 3 Therefore, additional description will be omitted to avoid redundancy.

[0149] Fig.15 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0150] Reference Fig.13 , Fig.14 and Fig.15 Describe how to perform HDR processing for three images with different fields of view. The image signal processor 1150 can generate the first image data IDAT1 of "J×K", the second image data IDAT2 of "M×N", and the third image data IDAT3 of "P×Q" based on the signals D1, D2, and D3 output from the first, second, and third image sensor blocks 1101, 1102, and 1103.

[0151] The main processor 1800 may generate fourth image data IDAT4 of “J×K” by cropping a region of the second image data IDAT2 corresponding to the first image data IDAT1. The main processor 1800 may generate fifth image data IDAT5 of “J×K” by cropping a region of the third image data IDAT3 corresponding to the first image data IDAT1.

[0152] Because the brightness values ​​of the first image data IDAT1, the second image data IDAT2, and the third image data IDAT3 are different from each other, the brightness values ​​of the first image data IDAT1, the fourth image data IDAT4, and the fifth image data IDAT5 may be different from each other. The main processor 1800 may perform HDR processing based on the first image data IDAT1, the fourth image data IDAT4, and the fifth image data IDAT5 having different brightness values. The main processor 1800 may obtain the image data IOUT through the HDR processing.

[0153] At the same time, Fig.15 In the embodiment of the present invention, the first image data IDAT1, the cropped fourth image data IDAT4, and the cropped fifth image data IDAT5 have the same resolution of "J×K". However, in a configuration in which the resolutions of the first image data IDAT1, the fourth image data IDAT4, and the fifth image data IDAT5 are different from each other, additional processing may be further required before HDR processing. Fig.16 Describes additional processing.

[0154] Fig.16 is a diagram illustrating an operation for HDR processing according to an embodiment.

[0155] For example, when the resolution of the first image data IDAT1 is greater than the resolution of the fifth image data IDAT5 and / or the resolution of the sixth image data IDAT6, the main processor 1800 may perform additional processing on the first image data IDAT1. For example, the main processor 1800 may obtain the fourth image data IDAT4 of "G×H" by performing merging on the first image data IDAT1 of "J×K". Therefore, the resolution of the fourth image data IDAT4, the resolution of the fifth image data IDAT5, and the resolution of the sixth image data IDAT6 may be equal.

[0156] At the same time, as well as reference Fig.16 In the case described above, there may be various instances where the resolution of the first image data IDAT1, the resolution of the fourth image data IDAT4, and the resolution of the fifth image data IDAT5 are different from each other, and various processing schemes for equally adjusting the resolutions may be applied to the respective cases. However, in various cases where the resolutions of the image data are different from each other, by Figures 7 to 9 The embodiments or combinations thereof may be sufficient to implement various processing schemes for equally adjusting resolutions. Therefore, additional descriptions will be omitted to avoid redundancy.

[0157] Fig.17 is a diagram illustrating the operation of an image sensor block according to a zoom factor. Fig.18 is a diagram illustrating the operation of the image sensor block in the first zoom mode. Fig.19 is a diagram illustrating the operation of the image sensor block in the second zoom mode.

[0158] Fig.17 Examples and Fig.13 The number of image sensor blocks is "3" (i.e., n=3). The zoom factor may be associated with the magnification or reduction (or enlargement or reduction) of an object. For example, a user may set a zoom condition for capturing a desired image by slowly or gradually increasing / decreasing the value of the zoom factor or selecting a specific value. Fig.15 In the example of , the zoom factor indicated by the x-axis may refer to the zoom value that the user wants. For example, the zoom factor may change between a minimum value ZF_min and a maximum value ZF_max. The minimum value ZF_min may be "1", but the inventive concept is not limited thereto. For example, the minimum value ZF_min may be a real number greater than "0" and less than "1". The maximum value ZF_max may be a real number greater than "1". The range of the zoom factor may include a threshold value ZF_th that causes a change in the zoom mode.

[0159] The user can use the user interface 1600 (refer to Fig.13 ) inputs a command for changing the zoom value to photograph objects at various ranges of distances. The main processor 1800 (reference Fig.13 ) can control the first to third image sensor blocks 1101 to 1103 (refer to Fig.13 ) operation.

[0160] refer to Fig.17 and Fig.18 The operation of the image processing block 1100 in the first zoom mode is described. In a configuration where the zoom factor that can be adjusted by the user is between the minimum value ZF_min and the threshold value ZF_th, the image processing block 1100 operates in the first zoom mode. In the first zoom mode, the first image sensor block 1101 may not operate, and the second image sensor block 1102 and the third image sensor block 1103 may operate. For example, the second image sensor block 1102 may capture an object corresponding to the region R2, and may output a second signal D2. The third image sensor block 1103 may capture an object in the region R3 and may output a third signal D3.

[0161] The image signal processor 1150 may generate the second image data IDAT2 based on the second signal D2, and may generate the third image data IDAT3 based on the third signal D3. For example, the image corresponding to the third image data IDAT3 may include the image corresponding to the second image data IDAT2. For example, the third image data IDAT3 of relatively high resolution may be associated with a pixel array of relatively high resolution, and the second image data IDAT2 of relatively medium resolution may be associated with a pixel array of relatively medium resolution. For example, the brightness value of the third image data IDAT3 may be greater than the brightness value of the second image data IDAT2.

[0162] The main processor 1800 may perform HDR processing on the second image data IDAT2 and the third image data IDAT3. The HDR processing may include referring to Figures 6 to 11 At least one of the described embodiments.

[0163] Meanwhile, although the user of the electronic device 1000 (refer to Fig.13 ) captures a video of an object, and the user can increase the zoom value to zoom in on the object. The zoom factor can be increased by increasing the zoom value. The increase in the zoom factor is conceptually illustrated as Fig.18 When the zoom factor reaches a certain threshold, the zoom mode of the image processing block 1100 may be changed from the first zoom mode to the second zoom mode.

[0164] refer to Fig.17 and Fig.19 The operation of the image processing block 1100 in the second zoom mode is described. In a configuration where the zoom factor that can be adjusted by the user is between the threshold value ZF_th and the maximum value ZF_max, the image processing block 1100 operates in the second zoom mode. In the second zoom mode, the third image sensor block 1103 may not operate, and the first image sensor block 1101 and the second image sensor block 1102 may operate. For example, the first image sensor block 1101 may capture an object in the region R1 and may output a first signal D1. The second image sensor block 1102 may capture an object in the region R2 and may output a second signal D2.

[0165] The image signal processor 1150 may generate first image data IDAT1 based on the first signal D1 and may generate second image data IDAT2 based on the second signal D2. For example, the image corresponding to the second image data IDAT2 may include the image corresponding to the first image data IDAT1. For example, the relatively medium-resolution second image data IDAT2 is associated with a relatively medium-resolution pixel array, and the relatively low-resolution first image data IDAT1 is associated with a relatively low-resolution pixel array. For example, the luminance value of the second image data IDAT2 may be greater than the luminance value of the first image data IDAT1.

[0166] The main processor 1800 may perform HDR processing on the first image data IDAT1 and the second image data IDAT2. The HDR processing may include referring to Figures 6 to 11 at least one of the embodiments described.

[0167] Fig. 20 is a block diagram illustrating Fig.13 an exemplary operation of the image sensor block.

[0168] The image processing block 1100 may include first to fifth image sensor blocks 1101, 1102, 1103, 1104, and 1105. The first image sensor block 1101 to the fifth image sensor block 1105 may include a first lens of the first field of view FOV1 to a fifth lens of the fifth field of view FOV5, respectively. The field of view of the first lens may be the smallest, and the field of view of the fifth lens may be the largest, and thus the field of view may increase across the lenses (i.e., FOV1 < FOV2 < FOV3 < FOV4 < FOV5). For example, the first lens may be a telephoto lens. The second lens and the third lens may be wide-angle lenses with different fields of view. The fourth lens and the fifth lens may be ultra-wide-angle lenses with different fields of view.

[0169] The first image sensor block 1101 may generate a first signal D1 associated with an image of a region R1 corresponding to the first field of view FOV1, the second image sensor block 1102 may generate a second signal D2 associated with an image of a region R2 corresponding to the second field of view FOV2, the third image sensor block 1103 may generate a third signal D3 associated with an image of a region R3 corresponding to the third field of view FOV3, the fourth image sensor block 1104 may generate a fourth signal D4 associated with an image of a region R4 corresponding to the fourth field of view FOV4, and the fifth image sensor block 1105 may generate a fifth signal D5 associated with an image of a region R5 corresponding to the fifth field of view FOV5.

[0170] Meanwhile, except that the image processing block includes five image sensor blocks, Fig. 20 the embodiment of Figure 5Therefore, additional description will be omitted to avoid redundancy.

[0171] Fig.21 is a diagram illustrating the operation of an image sensor block according to a zoom factor. Fig. 20 and Fig.21 Describes an image processing block that operates in multiple zoom modes.

[0172] In a configuration in which the zoom factor adjustable by the user is between the minimum value ZF_min and the first value ZF1, the image processing block 1100 operates in the first zoom mode. In the first zoom mode, the fourth and fifth image sensor blocks 1104 and 1105 operate, and the remaining first, second and third image sensor blocks 1101, 1102 and 1103 do not operate. The image signal processor 1150 generates image data based on the signals output from the fourth and fifth image sensor blocks 1104 and 1105. The main processor 1800 may perform reference processing based on the image data output from the image signal processor 1150. Figures 6 to 11 Describe the HDR process.

[0173] In a configuration in which the zoom factor has a value between the first value ZF1 and the second value ZF2 when the user increases the zoom magnification, the image processing block 1100 operates in the second zoom mode. In the second zoom mode, the third and fourth image sensor blocks 1103 and 1104 operate, and the remaining first, second, and fifth image sensor blocks 1101, 1102, and 1105 do not operate. The image signal processor 1150 generates image data based on signals output from the third and fourth image sensor blocks 1103 and 1104. The main processor 1800 may perform reference processing based on the image data output from the image signal processor 1150. Figures 6 to 11 Describe the HDR process.

[0174] Meanwhile, in a configuration in which the zoom factor has a value between the second value ZF2 and the third value ZF3 when the user continues to increase the zoom magnification, the image processing block 1100 operates in the third zoom mode. In the third zoom mode, the second and third image sensor blocks 1102 and 1103 operate, and the remaining first, fourth and fifth image sensor blocks 1101, 1104 and 1105 do not operate. In the third zoom mode, the main processor 1800 performs HDR processing based on the corresponding image data. When the zoom factor continues to increase, the image processing block 1100 may operate in the fourth zoom mode. In the above case, the main processor 1800 performs HDR processing based on the corresponding image data.

[0175] Fig. 22 is a diagram illustrating the operation of an image sensor block according to a zoom factor. Fig. 20 and Fig. 22Describes an image processing block that operates in multiple zoom modes.

[0176] In a configuration in which the zoom factor adjustable by the user is between the minimum value ZF_min and the first value ZF1, the image processing block 1100 operates in the first zoom mode. In the first zoom mode, the third, fourth, and fifth image sensor blocks 1103, 1104, and 1105 operate, and the remaining first and second image sensor blocks 1101 and 1102 do not operate. The image signal processor 1150 generates image data based on signals output from the third, fourth, and fifth image sensor blocks 1103, 1104, and 1105. The main processor 1800 may perform reference processing based on the image data output from the image signal processor 1150. Fig.15 and Fig.16 At least one of the multiple HDR processes described.

[0177] In a configuration in which the zoom factor has a value between the first value ZF1 and the second value ZF2 when the user increases the zoom magnification, the image processing block 1100 operates in the second zoom mode. In the second zoom mode, the second, third, and fourth image sensor blocks 1102, 1103, and 1104 operate, and the remaining first and fifth image sensor blocks 1101 and 1105 do not operate. The image signal processor 1150 generates image data based on signals output from the second, third, and fourth image sensor blocks 1102, 1103, and 1104. The main processor 1800 may perform HDR processing based on the image data output from the image signal processor 1150.

[0178] In a configuration in which the zoom factor has a value between the second value ZF2 and the maximum value ZF_max when the user continues to increase the zoom magnification, the image processing block 1100 operates in a third zoom mode. In the third zoom mode, the first, second, and third image sensor blocks 1101, 1102, and 1103 operate, and the remaining fourth and fifth image sensor blocks 1104 and 1105 do not operate. In the third zoom mode, the main processor 1800 performs HDR processing based on the corresponding image data.

[0179] Meanwhile, the description is made under the assumption that the number of image sensor blocks is 5. Figure 20 to Figure 22 That is, additional image sensor blocks may be used, and reference may be made to Figure 20 to Figure 22 The described techniques may be equally applied to configurations employing six or more image sensor blocks.

[0180] Fig.23 The diagram depends on Fig.13 A conceptual diagram of an exemplary method for adjusting a magnification based on a command received from a user interface.

[0181] In some cases, the user 30 may want to input the zoom magnification to the user interface 1600 while slowly increasing or decreasing the value of the zoom magnification. For example, two fingers of the user 30 may touch the touch screen 1610 of the user interface 1600. Next, the user 30 may take an action of widening or narrowing the distance between the two fingers to slowly adjust the value of the zoom magnification (e.g., pinch zoom-in or pinch zoom-out).

[0182] For another example, the user 30 may use the graphical interface B1 displayed on the display device of the user interface 1600 to adjust the value of the zoom ratio. The user 30 may slowly adjust the value of the zoom ratio by sliding their finger along the slide bar of the graphical interface B1. In another example, the user 30 may press the button 1620 of the user interface 1600 to slowly increase or decrease the value of the zoom ratio.

[0183] However, the operation of adjusting the zoom ratio by touch is exemplary and is not intended to limit the inventive concept. For example, the zoom ratio may be adjusted by inputting a command through a user's voice, gesture, gesture using a pen, etc.

[0184] According to the embodiment, by obtaining image data to be used for HDR (High Dynamic Range) processing via a plurality of image sensors, HDR processing may be efficiently performed.

[0185] While the inventive concept has been described with reference to exemplary embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications may be made thereto without departing from the spirit and scope of the inventive concept as set forth in the following claims.

Claims

1. An electronic device for processing a plurality of video frames, comprising: a first image sensor block configured to generate a first data signal by photographing an object with a first field of view, and sequentially generate a third data signal by photographing the object with the first field of view; a second image sensor block configured to generate a second data signal by photographing the object with a second field of view larger than the first field of view, and sequentially generate a fourth data signal by photographing the object with the second field of view; an image signal processor configured to generate first image data of a first resolution based on the first data signal, generate second image data of a second resolution based on the second data signal, generate third image data of a third resolution based on the third data signal, and generate fourth image data of a fourth resolution based on the fourth data signal; and Main processor, configured as: obtaining first cropped image data from among the second image data corresponding to the entire first image data, obtaining second cropped image data from among the fourth image data corresponding to the entire first image data or the entire third image data, and performing high dynamic range (HDR) processing based on the first image data, the third image data, the first cropped image data, and the second cropped image data to generate an HDR processed image, The HDR processed image is the first video frame among multiple video frames.

2. The electronic device according to claim 1, wherein: When the first resolution is greater than a resolution of the first cropped image data, the main processor is further configured to perform merging on the first image data to obtain merged image data.

3. The electronic device as claimed in claim 2, wherein: The resolution of the merged image data is equal to the resolution of the first cropped image data.

4. The electronic device according to claim 1, wherein: When the first resolution is less than a resolution of the first cropped image data, the main processor is further configured to perform merging on the first cropped image data to obtain merged image data.

5. The electronic device as claimed in claim 4, wherein: The first resolution is equal to a resolution of the merged image data.

6. The electronic device as claimed in claim 1, wherein: When the first resolution is less than a resolution of the first cropped image data, the main processor is further configured to perform resolution upscaling on the first image data to obtain resolution-upscaled image data.

7. The electronic device as claimed in claim 6, wherein: The resolution of the increased-resolution image data is equal to the resolution of the first cropped image data.

8. The electronic device as claimed in claim 1, wherein: When the first resolution and the third resolution are greater than resolutions of the first cropped image data and the second cropped image data, the main processor is further configured to perform merging on the first image data and the third image data to obtain merged image data.

9. The electronic device as claimed in claim 1, wherein: The first image sensor block includes a first pixel array of a first plurality of pixels, wherein the second image sensor block comprises a second pixel array of a second plurality of pixels, and The second number of the second plurality of pixels is greater than the first number of the first plurality of pixels.

10. The electronic device according to claim 1, wherein: The resolution of the first cropped image data is half or less than the second resolution.

11. An electronic device for processing a plurality of video frames, comprising: a first image sensor block configured to capture an object with a first field of view and generate a first data signal corresponding to the object captured with the first field of view; a second image sensor block configured to capture the object with a second field of view larger than the first field of view, and to generate a second data signal corresponding to the object captured with the second field of view; a third image sensor block configured to photograph the object with a third field of view larger than the second field of view, and to generate a third data signal corresponding to the object photographed with the third field of view; an image signal processor configured to generate first image data of a first resolution based on the first data signal, generate second image data of a second resolution based on the second data signal, and generate third image data of a third resolution based on the third data signal; and Main processor, configured as: performing high dynamic range (HDR) processing based on the first image data, the second image data and the third image data and in a first zoom factor period, obtaining cropped image data from among the third image data corresponding to the entire second image data, and performing HDR processing based on the second image data and the cropped image data to generate an HDR processed image, The HDR processed image is the first video frame among the multiple video frames.

12. The electronic device according to claim 11, wherein: When the second resolution is greater than a resolution of the cropped image data, the main processor is further configured to perform merging on the second image data to obtain merged image data.

13. The electronic device according to claim 11, wherein: In a second zoom factor mode greater than the first zoom factor period, the main processor is further configured to obtain second cropped image data from the second image data corresponding to the entire first image data, and perform HDR processing based on the first image data and the second cropped image data.

14. The electronic device as claimed in claim 13, wherein: When the first resolution is greater than a resolution of the second cropped image data, the main processor is further configured to perform merging on the first image data to obtain merged image data.

15. A method of processing a plurality of video frames by using a first image sensor block having a first field of view and a second image sensor block having a second field of view larger than the first field of view, the method comprising: generating, by the first image sensor block, a first data signal by photographing an object with the first field of view, and sequentially generating a third data signal by photographing the object with the first field of view; generating, by the second image sensor block, a second data signal by photographing the object with the second field of view, and sequentially generating a fourth data signal by photographing the object with the second field of view; generating first image data of a first resolution based on the first data signal; generating second image data of a second resolution based on the second data signal; generating third image data of a third resolution based on the third data signal; generating fourth image data of a fourth resolution based on the fourth data signal; generating first cropped image data from among the second image data corresponding to the entire first image data; generating second cropped image data from among the fourth image data corresponding to the entire first image data or the entire third image data; and performing high dynamic range (HDR) processing based on the first image data, the third image data, the first cropped image data, and the second cropped image data to generate an HDR processed image, The HDR processed image is a first video frame of multiple video frames.

16. The method of claim 15, further comprising: The first resolution and a resolution of the first cropped image data are compared.

17. The method of claim 16, further comprising: When the first resolution is greater than the resolution of the first cropped image data: performing merging on the first image data to obtain merged image data; and The HDR processing is performed based on the merged image data and the first cropped image data.

18. The method of claim 16, further comprising: When the first resolution is smaller than the resolution of the first cropped image data: performing merging on the first cropped image data to obtain merged image data; and The HDR processing is performed based on the first image data and the merged image data.

19. The method of claim 16, further comprising: When the first resolution is smaller than the resolution of the first cropped image data: performing resolution enhancement on the first image data to obtain image data with enhanced resolution; and The HDR processing is performed based on the resolution increased image data and the first cropped image data.

Images