Application processor, electronic device including the same, and method of operating an electronic device

Abstract

To provide an application processor, an electronic device including the same, and an operation method of the electronic device.SOLUTION: An electronic device includes: a display; a first camera module having a first field of view and generating first image data; a second camera module having a second field of view that is less than the first field of view and generating second image data; and an application processor configured to obtain zoom information including a zoom ratio and a region of interest determined based on a user's input, to generate converted image data by scaling an image area corresponding to the region of interest with respect to image data corresponding to the zoom ratio from among the first image data and the second image data, and to control the display to display the converted image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an application processor, an electronic device including the same, and an operating method of the electronic device, and more particularly to an application processor that performs a zoom function using multiple camera modules having different Field of View (FoV), an electronic device including the same, and an operating method of the electronic device. [Background technology]

[0002] In recent years, the demand for camera modules including image sensors has been increasing, and in particular, the demand for multi-camera systems including multiple camera modules has been continuously increasing. For example, a smartphone equipped with two or more camera modules having different viewing angles can display images with various zoom magnifications using the two or more camera modules.

[0003] Meanwhile, in a conventional multi-camera system, when a zoom function is performed on an image, digital zoom is performed based on the center point of the frame. As a result, if the object that a user wants to view is not adjacent to the center point of the frame, the user must directly adjust the angle or position of the multi-camera system so that the object is located at the center point of the frame. In particular, if the multi-camera system performs a high-speed zoom function, the viewing angle becomes very narrow. Therefore, even a small change in the angle or position of the multi-camera system can cause a sudden change in the viewing angle, making it difficult for the user to adjust the multi-camera system. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an application processor that enlarges or reduces image data output by multiple camera modules based on a zoom factor and a region of interest determined based on user input, an electronic device including the same, and a method of operating the electronic device. [Means for solving the problem]

[0005] To achieve the above object, an electronic device according to one aspect of the technical idea of ​​the present invention includes a display, a first camera module having a first viewing angle and generating first image data, a second camera module having a second viewing angle narrower than the first viewing angle and generating second image data, and an application processor that acquires zoom information including a zoom factor and a region of interest determined based on a user input, scales an image region corresponding to the region of interest for image data corresponding to the zoom factor among the first image data and the second image data, generates transformed image data, and controls the display to display the transformed image data.

[0006] According to one aspect of the technical idea of ​​the present invention, an application processor includes a first image signal processor that receives first image data captured at a first viewing angle from a first camera module, performs image processing based on the first image data, and generates a first image signal; a second image signal processor that receives second image data captured at a second viewing angle narrower than the first viewing angle from a second camera module, performs image processing based on the second image data, and generates a second image signal; and a third image signal processor that generates converted image data based on at least one of the first image signal and the second image signal. The image signal processor acquires zoom information including a zoom factor and a region of interest determined based on a user input, and the first image signal processor scales an image region corresponding to the region of interest for the first image data and generates the first image signal if the zoom factor is within a first zoom factor range, and the second image signal processor scales an image region corresponding to the region of interest for the second image data and generates the second image signal if the zoom factor is within a second zoom factor range, and the third image signal processor selects at least one of the first image signal and the second image signal according to the zoom factor and generates converted image data using the selected image signal.

[0007] According to one aspect of the technical concept of the present invention, a method for operating an electronic device includes receiving first image data captured at a first viewing angle from a first camera module, receiving second image data captured at a second viewing angle from a second camera module, calculating a zoom factor and a region of interest based on a user's touch gesture, scaling a region corresponding to the region of interest in the first image data if the zoom factor is within a first zoom factor range, and scaling a region corresponding to the region of interest in the second image data if the zoom factor is within a second zoom factor range, and generating converted image data based on at least one of the scaled first image data and the scaled second image data. [Effects of the Invention]

[0008] According to an application processor, an electronic device including the same, and an operating method of an electronic device according to embodiments of the present invention, the viewing angle of a camera module (e.g., a telephoto camera module) corresponding to a high zoom ratio can be pre-adjusted to correspond to a region of interest, thereby reducing the delay required to display an image corresponding to a region of interest when switching camera modules due to an increase in zoom ratio. Also, it is possible to prevent the inconvenience of a user having to manually adjust the angle of the camera module so that the region of interest is included in the viewing angle of the camera module corresponding to a high zoom ratio. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an electronic device according to one embodiment of the present invention. [Figure 2] 1 is a diagram illustrating a specific configuration of a camera module according to an embodiment of the present invention; [Figure 3] 1 is a diagram illustrating an image capturing operation of an electronic device according to an exemplary embodiment of the present invention; [Figure 4]1 is a diagram illustrating an operation of an image processing device according to an embodiment of the present invention; [Figure 5] 10 is a flowchart illustrating a zoom parameter update operation of a zoom information generator according to an embodiment of the present invention. [Figure 6] 1 is a diagram illustrating touch gestures associated with a zoom function according to an embodiment of the present invention; [Figure 7] 1 is a diagram illustrating a method for calculating a region of interest according to an embodiment of the present invention; [Figure 8] 10 is a diagram illustrating touch gestures associated with a zoom function according to another embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating a control signal generation operation of a camera module controller according to an embodiment of the present invention. [Figure 10] 2 is a block diagram showing a specific configuration of a transformed image generator according to an embodiment of the present invention; [Figure 11] 3 is a flowchart illustrating an operation of an image processing device according to an embodiment of the present invention. [Figure 12A] FIG. 2 is a sequence diagram illustrating the operations between components of an electronic device to perform a zoom function. [Figure 12B] FIG. 2 is a sequence diagram illustrating the operations between components of an electronic device to perform a zoom function. [Figure 12C] FIG. 2 is a sequence diagram illustrating the operations between components of an electronic device to perform a zoom function. [Figure 12D] FIG. 2 is a sequence diagram illustrating the operations between components of an electronic device to perform a zoom function. [Figure 12E] FIG. 2 is a sequence diagram illustrating the operations between components of an electronic device to perform a zoom function. [Figure 13A] 10 is a diagram showing a result of scaling image data according to a zoom magnification and an output image on a display. [Figure 13B] 10 is a diagram showing a result of scaling image data according to a zoom magnification and an output image on a display. [Figure 14A] 1 is a diagram illustrating an image displayed on a display at a high zoom ratio according to an embodiment of the present invention. [Figure 14B] 10 is a diagram illustrating an operation of selecting a region of interest from a user at a high zoom ratio according to an embodiment of the present invention. [Figure 15A] 1 is a block diagram of an electronic device including an application processor according to one embodiment of the present invention. [Figure 15B] 1 is a block diagram of an electronic device including an application processor according to one embodiment of the present invention. [Figure 15C] 1 is a block diagram of an electronic device including an application processor according to one embodiment of the present invention. [Figure 16A] 1 is a diagram illustrating an image capturing operation of an electronic device according to an exemplary embodiment of the present invention; [Figure 16B] 1 is a diagram illustrating an image capturing operation of an electronic device according to an exemplary embodiment of the present invention; [Figure 16C] 1 is a diagram illustrating an image capturing operation of an electronic device according to an exemplary embodiment of the present invention; [Figure 17] 1 is a block diagram showing a specific configuration of an application processor according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a block diagram illustrating an electronic device 1000 according to one embodiment of the present invention.

[0011] 1, electronic device 1000 includes a first camera module 1100a, a second camera module 1100b, an image processing device 1200, a display 1300, an input device 1400, and / or a memory 1500. As an example, electronic device 1000 may be embodied as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device, such as a laptop computer, a mobile phone, a smartphone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, or a wearable device.

[0012] The first camera module 1100a and the second camera module 1100b may each capture an image of a subject (or object) outside the electronic device 1000 and generate image data. For example, the first camera module 1100a may include a first image sensor 1110a, and the second camera module 1100b may include a second image sensor 1110b. The first image sensor 1110a and the second image sensor 1110b may each convert an optical signal of the subject into an electrical signal using an optical lens (not shown). To this end, the first image sensor 1110a and the second image sensor 1110b may include a pixel array in which a plurality of pixels are two-dimensionally arranged. For example, each of the plurality of pixels may be assigned one of a plurality of reference colors. For example, the plurality of reference colors may include RGB (red, green, blue) or RGBW (red, green, blue, white). As non-limiting examples, the first image sensor 1110a and the second image sensor 1110b can be implemented using a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS), or can be implemented as various other types of photoelectric conversion elements.

[0013] The first camera module 1100a may generate first image data IDT1 using the first image sensor 1110a, and the second camera module 1100b may generate second image data IDT2 using the second image sensor 1110b. The first image data IDT1 and the second image data IDT2 are variously referred to as image frames and frame data. The first camera module 1100a and the second camera module 1100b may transmit the generated first image data IDT1 and second image data IDT2 to the image processing device 1200, respectively.

[0014] The first camera module 1100a and the second camera module 1100b may each generate new first image data IDT1 or second image data IDT2 at predetermined time intervals, and the frequency at which new image data is generated is referred to as the frame rate. That is, the frame rate may represent the number of newly generated image data per unit time.

[0015] The first camera module 1100a and the second camera module 1100b may have different field of view (FoV). For example, the second field of view of the second camera module 1100b may be narrower than the first field of view of the first camera module 1100a. For example, the first camera module 1100a may be a wide-angle camera, and the second camera module 1100b may be a telephoto camera. Therefore, for the same object, the first camera module 1100a may generate first image data IDT1 with a wide angle, and the second camera module 1100b may generate second image data IDT2 with a narrow angle.

[0016] In particular, the second camera module 1100b is also embodied as a folded camera including a prism 1105, as described below in FIG. 2, and the second field of view angle of the second camera module 1100b can be changed by adjusting the position of the prism 1105.

[0017] The image processing device 1200 can perform an image processing operation on at least one of the first image data IDT1 and the second image data IDT2 received from the first camera module 1100a and the second camera module 1100b to generate converted image data CDT.

[0018] The display 1300 can display various content (e.g., text, images, videos, icons or symbols, etc.) to a user based on the converted image data CDT received from the image processing device 1200. For example, the display 1300 can include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display.

[0019] The input device 1400 may transmit commands or data input from a user or other external devices to other components of the electronic device 1000, or may output commands or data received from other components of the electronic device 1000 to the user or other external devices. In one example, the input device 1400 may be embodied as a touch panel and may sense a user's touch input. For example, the input device 1400, which is a touch panel, may sense at least one touch input through a user's body (e.g., a finger) or a dedicated touch input means. The input device 1400 may convert an analog signal corresponding to the touch input into a digital signal (e.g., coordinate information such as an X coordinate and a Y coordinate) and provide the digital signal to the image processing device 1200 as a user input UI. In one example, when the input device 1400 is a touch panel, the input device 1400 and the display 1300 may be embodied as a single device. For convenience of explanation, the present specification will be described assuming that the input device 1400 is embodied as a touch panel. However, it goes without saying that the technical idea of ​​the present invention is not limited thereto and can be realized by various devices such as a keyboard or a mouse.

[0020] The memory 1500 is a storage location for storing data, and may store, for example, an operating system (OS), various programs, and various data (e.g., image data). The memory 1500 may be a volatile memory such as a dynamic random access memory (DRAM) or a static RAM (SRAM), or a non-volatile memory such as a phase change RAM (PRAM), a resistive RAM (ReRAM), or a flash memory. In one example, the image processing device 1200 may store converted image data CDT in the memory 1500. The converted image data CDT stored in the memory 1500 may be subsequently used by the image processing device 1200 or may be stored in a separate storage device.

[0021] According to an embodiment of the present invention, the image processing device 1200 may provide a zoom function for enlarging or reducing an image displayed on the display 1300. In one example, the image processing device 1200 may enlarge or reduce an image using a digital zoom method or an optical zoom method and display it on the display 1300. To this end, the image processing device 1200 may include a conversion image generator 1210, a camera module controller 1230, and / or a zoom information generator 1250.

[0022] The converted image generator 1210 can generate the converted image data CDT based on the first image data IDT1 and the second image data IDT2 received from the first camera module 1100a and the second camera module 1100b. In one example, the converted image generator 1210 can generate the converted image data CDT by cropping and scaling (enlarging or reducing) the first image data IDT1 and / or the second image data IDT2 based on a zoom parameter.

[0023] Here, the zoom parameters may include various parameters related to the zoom function. For example, the zoom parameters may include a target zoom factor desired by the user and a region of interest that the user desires to enlarge or reduce. The region of interest may be divided into pixel units in the image area, or into regions larger than pixel units. For example, the region of interest may be divided into rectangles having a preset aspect ratio, which may correspond to the aspect ratio of the display 1300 (e.g., 16:9 or 2:1). Meanwhile, the types of zoom parameters are not limited to the above examples and may include various other parameters.

[0024] Specifically, the converted image generator 1210 may select image data corresponding to a target zoom magnification from among the first image data IDT1 and the second image data IDT2, and may generate converted image data CDT by enlarging or reducing an image region corresponding to a region of interest for the selected image data by the target zoom magnification.

[0025] The camera module controller 1230 can control the first camera module 1100a and the second camera module 1100b. In one example, the camera module controller 1230 can generate a control signal CTRL to control the second camera module 1100b, which has a narrower field of view than the first camera module 1100a, based on the zoom parameters. For example, the camera module controller 1230 can generate a control signal CTRL to control the second camera module 1100b to change the second field of view angle to a field of view corresponding to the region of interest included in the zoom parameters. In response to the received control signal CTRL, the second camera module 1100b can adjust the position of the prism 1105 so that it has a field of view corresponding to the region of interest.

[0026] On the other hand, although FIG. 1 shows that the camera module controller 1230 transmits the control signal CTRL only to the second camera module 1100b, the technical idea of ​​the present invention is not limited thereto, and it goes without saying that the camera module controller 1230 can also transmit the control signal CTRL to the first camera module 1100a.

[0027] The zoom information generator 1250 can calculate each zoom parameter based on the user input and update the zoom information including the zoom parameters based on the calculated zoom parameters. In one example, the zoom information generator 1250 can receive a user input UI through the input device 1400, calculate a target zoom magnification and a region of interest based on the received user input UI, and update the existing zoom information.

[0028] For example, if the input device 1400 is a touch panel, the user input UI may include a touch gesture. Here, the touch gesture may be a single touch input or continuous touch inputs. The touch gesture may include various gestures, such as a touch, a tap, a double tap, a long tap, a drag, a drag-and-drop, a flick, a swipe, and a pinch. The input device 1400 may provide coordinate information of the touch gesture to the zoom information generator 1250. The zoom information generator 1250 may calculate a target zoom factor desired by the user and a region of interest that the user desires to enlarge or reduce based on the coordinate information of the touch gesture.

[0029] The zoom information generator 1250 can then provide the zoom information to the transformed image generator 1210 and the camera module controller 1230. The transformed image generator 1210 and the camera module controller 1230 can perform their respective functions using the zoom parameters included in the received zoom information.

[0030] The camera module controller 1230 and the zoom information generator 1250 may each be implemented as software, hardware, or a combination of software and hardware, such as firmware. When the zoom information generator 1250 or the camera module controller 1230 is implemented as software, the respective functions described above are implemented as programmed source code and loaded into a memory provided in the image processing device 1200 or stored in a read-only memory (ROM). A processor (e.g., a microprocessor) provided in the image processing device 1200 executes the software to implement the functions of the zoom information generator 1250 or the camera module controller 1230. When the zoom information generator 1250 or the camera module controller 1230 is implemented as hardware, the zoom information generator 1250 or the camera module controller 1230 may include a logic circuit and a register, and may perform the respective functions described above based on the register settings.

[0031] 2 is a diagram illustrating a specific configuration of a camera module according to an embodiment of the present invention. Hereinafter, the detailed configuration of the second camera module 1100b will be described in detail with reference to FIG. 2. However, the following description may also be applied to the first camera module 1100a, depending on the embodiment.

[0032] Referring to FIG. 2, the second camera module 1100b includes a prism 1105, an optical path folding element (OPFE) 1120, an actuator 1130, an image sensing device 1140, and / or a storage 1150.

[0033] The prism 1105 includes a reflective surface 1107 made of a light-reflecting material, and can change the path of light L incident from the outside. In one example, the prism 1105 can change the path of light L incident in a first direction X to a second direction Y. In this case, the angle of incidence of the light L incident on the reflective surface 1107 from the first direction X is the same as the angle of reflection of the light L reflected from the reflective surface 1107 in the second direction Y.

[0034] The prism 1105 can rotate the reflective surface 1107 of the light-reflecting material in direction A around the central axis 1106, or rotate the central axis 1106 itself in direction B, thereby changing the path of the light L incident in the first direction X to the second direction Y. At this time, the OPFE 1120 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0035] In one example, as shown, the maximum rotation angle of the prism 1105 in the A direction is 15° or less in the plus (+) A direction and greater than 15° in the minus (-) A direction, but the embodiment is not limited thereto. In one example, the prism 1105 can rotate in the plus (+) or minus (-) B direction by 20° or less, 10° to 20°, or 15° to 20°, where the rotation angle is the same in the plus (+) or minus (-) B direction, or a similar angle within a range of 1° or less. In one example, the prism 1105 can rotate the reflective surface 1107 of the light-reflecting material in a third direction (e.g., Z direction) parallel to the extension direction of the central axis 1106.

[0036] The OPFE 1120 includes, for example, m optical lenses (where m is a positive integer) in groups. The m lenses can move in the second direction Y to change the optical zoom magnification of the second camera module 1100b. For example, if the basic optical zoom magnification of the second camera module 1100b is Z, when the m optical lenses included in the OPFE 1120 are moved, the optical zoom magnification of the second camera module 1100b can be changed to an optical zoom magnification of 3Z, 5Z, or more than 5Z.

[0037] The actuator 1130 can move the OPFE 1120 or the optical lens to a specific and / or desired position. For example, the actuator 1130 can adjust the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for accurate sensing. The actuator 1130 can then adjust the position of the prism 1105.

[0038] 1, the actuator 1130 receives a control signal CTRL for controlling the second camera module 1100b to change the viewing angle, and can adjust the position of the prism 1105 based on the received control signal CTRL. For example, the actuator 1130 can use the control signal CTRL to rotate the reflecting surface 1107 of the prism 1105 in direction A, move the central axis 1106 of the prism 1105 in direction B, or move the reflecting surface 1107 of the prism 1105 in direction Z. This allows the viewing angle of the second camera module 1100b to be changed in the up, down, left, and right directions.

[0039] The image sensing device 1140 includes an image sensor 1110b, control logic 1142, and / or memory 1144. The image sensor 1110b may sense an image of a sensing target using light L provided through an optical lens. The control logic 1142 may control the overall operation of the second camera module 1100b. For example, the control logic 1142 may control the operation of the second camera module 1100b according to a control signal CTRL provided from the camera module controller 1230.

[0040] The memory 1144 may store information necessary for the operation of the second camera module 1100b, such as calibration data 1146. The calibration data 1146 includes information necessary for the second camera module 1100b to generate second image data IDT2 using externally provided light L. The calibration data 1146 may include, for example, information related to the rotation angle, focal length, and optical axis. If the second camera module 1100b is implemented as a multi-state camera in which the focal length changes depending on the position of the optical lens, the calibration data 1146 may include focal length values ​​for each position (or state) of the optical lens and information related to autofocusing.

[0041] The storage 1150 can store second image data IDT2 sensed through the second image sensor 1110b. The storage 1150 can be disposed outside the image sensing device 1140, and can also be implemented in a form stacked with a sensor chip constituting the image sensing device 1140. In some embodiments, the storage 1150 is implemented using an EEPROM (Electrically Erasable Programmable Read-Only Memory), but the embodiment is not limited thereto.

[0042] 1 includes an actuator 1130, as does the second camera module 1100b, such that the first camera module 1100a and the second camera module 1100b each include the same or different calibration data 1146 due to the operation of the actuator 1130 included therein.

[0043] In one example, the first camera module 1100a, like the second camera module 1100b, may be a folding camera-type camera module including the prism 1105 and OPFE 1120. Alternatively, the first camera module 1100a may be a vertical camera-type camera module that does not include the prism 1105 and OPFE 1120. Alternatively, the first camera module 1100a may be a vertical depth camera that has a vertical camera type and extracts depth information using infrared rays (IR). In this case, the image processing device 1200 may merge image data provided from the depth camera with image data provided from another camera module (e.g., the second camera module 1100b) to generate a 3D depth image. However, the type of the first camera module 1100a is not limited to the above example.

[0044] FIG. 3 is a diagram illustrating an image capturing operation of the electronic device 1000 according to an embodiment of the present invention.

[0045] 3, the electronic device 1000 may acquire first image data IDT1 and second image data IDT2 by capturing an image of an object using a first camera module 1100a and a second camera module 1100b. In one example, the electronic device 1000 may acquire first image data IDT1 of the object through the first camera module 1100a having a wide angle, and acquire second image data IDT2 of the object through the second camera module 1100b having a narrow angle. The first image data IDT1 may have a wider image area than the second image data IDT2, and the first image data IDT1 and the second image data IDT2 may have overlapping areas.

[0046] According to an embodiment of the present invention, the second camera module 1100b can change the viewing angle by adjusting the position of the prism 1105 according to a control signal CTRL from the camera module controller 1230. If the viewing angle of the second camera module 1100b is changed, the overlapping area between the first image data IDT1 and the second image data IDT2 can also be changed.

[0047] Figure 4 is a diagram illustrating the operation of the image processing device 1200 according to an embodiment of the present invention. More specifically, Figure 4 is a diagram illustrating the operation of the zoom function of the image processing device 1200 of Figure 1. Hereinafter, the zoom-in operation of the image processing device 1200, which enlarges an image, will be described as a zoom function. However, the following description may also be applied to the zoom-out operation of reducing an image, depending on the embodiment.

[0048] 1 and 4, by capturing an image IMG of the same subject, a first camera module 1100a can acquire first image data IDT1 with a wide angle, and a second camera module 1100b can acquire second image data IDT2 with a narrow angle. The first camera module 1100a and the second camera module 1100b can output the acquired first image data IDT1 and second image data IDT2 to a conversion image generator 1210, respectively.

[0049] The transformed image generator 1210 generates transformed image data CDT by performing image enlargement using the zoom parameters on at least one of the first image data IDT1 and the second image data IDT2. In this case, the transformed image generator 1210 can obtain the zoom parameters by receiving zoom information Info_Z from the zoom information generator 1250.

[0050] In one example, the transformed image generator 1210 may select image data corresponding to a target zoom magnification from among the first image data IDT1 and the second image data IDT2. Then, the transformed image generator 1210 may generate transformed image data CDT having the target zoom magnification by enlarging an image region corresponding to a region of interest in the selected image data. In one example, the transformed image generator 1210 may crop an image region corresponding to the region of interest from the selected image data to generate a cropped region, and scale the cropped region to generate transformed image data CDT having the target zoom magnification.

[0051] The correspondence between the target zoom magnification and the image data is also based on the characteristics (e.g., field of view) of the camera module that generated the image data. For example, a first zoom magnification range including a relatively low target zoom magnification (e.g., 1.0x or more and less than 5.0x) can be set to correspond to the first image data IDT1 generated by the first camera module 1100a having a wide angle. A second zoom magnification range including a relatively high target zoom magnification (e.g., 5.0x or more) can be set to correspond to the second image data IDT2 generated by the second camera module 1100b having a narrow angle. However, the present invention is not limited to the above example, and the first and second zoom magnification ranges can also be embodied as overlapping zoom magnification ranges.

[0052] Therefore, the converted image generator 1210 generates converted image data CDT with an enlarged region of interest based on the first image data IDT1 generated by the first camera module 1100a in the first zoom magnification range, and generates converted image data CDT with an enlarged region of interest based on the second image data IDT2 generated by the second camera module 1100b in the second zoom magnification range. That is, camera module switching can be performed at the boundary between the first zoom magnification range and the second zoom magnification range.

[0053] In one example, the first camera module 1100a and the second camera module 1100b can acquire new first image data IDT1 and new second image data IDT2 at a preset frame rate and output them to the converted image generator 1210. Therefore, the converted image generator 1210 can periodically receive new first image data IDT1 and new second image data IDT2 and repeat the above-described operations using the received new first image data IDT1 and new second image data IDT2.

[0054] The zoom information generator 1250 generates zoom information Info_Z including zoom parameters such as a target zoom magnification and a region of interest, and can provide the generated zoom information Info_Z to the transformed image generator 1210 and the camera module controller 1230. The transformed image generator 1210 and the camera module controller 1230 may each internally include a buffer (not shown) for storing the zoom information Info_Z.

[0055] In one example, the zoom information generator 1250 may first generate zoom information Info_Z including a target zoom factor having a default zoom factor value and a region of interest having a default region value. In a non-limiting example, the default zoom factor may be set to a value equal to the default optical zoom factor (e.g., 1.0x) of the first camera module. The default region may be set to a rectangular region that includes the center point of the image region and has the screen ratio of the display 1300, e.g., a value equal to the entire image region of the first image data IDT1.

[0056] Therefore, when the converted image generator 1210 starts a zoom function, it can receive zoom information Info_Z having default values ​​and generate converted image data CDT based on the zoom parameters having default values ​​included in the received zoom information Info_Z. Specifically, the converted image generator 1210 can select the first image data IDT1 as image data corresponding to the default zoom factor. Then, the converted image generator 1210 can crop an image area corresponding to the default area of ​​the first image data IDT1, generate a cropped area, and scale the cropped area to generate converted image data CDT having the default zoom factor. The generated converted image data CDT can then be displayed on the display 1300. Meanwhile, as in the previous example, if the default area is set to the same value as the entire image area of ​​the first image data IDT1, the converted image generator 1210 can generate converted image data CDT from the first image data IDT1 without cropping or scaling.

[0057] Meanwhile, unlike the above example, the transformed image generator 1210 may be implemented to store zoom parameters having default values ​​in an internal buffer (not shown) in advance. In this case, the transformed image generator 1210 can generate transformed image data CDT using the previously stored zoom parameters even if it has not yet received zoom information Info_Z from the zoom information generator 1250.

[0058] In one example, the zoom information generator 1250 can update the zoom parameters based on the user input UI received from the input device 1400. Then, the zoom information generator 1250 can generate new zoom information Info_Z based on the updated zoom parameters and provide the new zoom information Info_Z to the transformed image generator 1210 and the camera module controller 1230.

[0059] Specifically, the zoom information generator 1250 may receive a user input UI including coordinate information of a touch gesture from the input device 1400 while the converted image data CDT generated based on the zoom parameters having the default values ​​is displayed. Then, the zoom information generator 1250 may determine whether the touch gesture is associated with a zoom function based on the coordinate information. If the touch gesture is associated with a zoom function, the zoom information generator 1250 may update the zoom parameters based on the user input UI.

[0060] Touch gestures associated with the zoom function include at least one of a touch gesture corresponding to zooming in, a touch gesture corresponding to zooming out, and a touch gesture for selecting a region of interest. In a non-limiting example, a touch gesture corresponding to zooming in can be set to a pinch-out, which increases the distance between two fingers (or a dedicated touch input means) while they are touching each other, and a touch gesture corresponding to zooming out can be set to a pinch-in, which decreases the distance between the fingers. Furthermore, touch gestures for selecting a region of interest can be set to not only the pinch-out and pinch-in mentioned above, but also touch, tap, double-tap, long tap, drag-and-drop, pinch-drag, etc.

[0061] When the zoom information generator 1250 confirms a touch gesture corresponding to a zoom function, it may calculate a target zoom magnification and / or a region of interest based on coordinate information. For example, if the touch gesture is a pinch out, the zoom information generator 1250 may calculate a target zoom magnification and a region of interest based on coordinate information. As another example, if the touch gesture is a tap, the zoom information generator 1250 may calculate a region of interest based on coordinate information. Meanwhile, specific operations of the zoom information generator 1250 for calculating the zoom magnification and the region of interest will be described later with reference to FIGS. 5 to 8.

[0062] When the converted image generator 1210 receives new zoom information Info_Z from the zoom information generator 1250, the converted image generator 1210 can generate new converted image data CDT using the target zoom magnification and region of interest included in the new zoom information Info_Z. Specifically, since the first camera module 1100a and the second camera module 1100b output new first image data IDT1 and new second image data IDT2 at a preset period, the converted image generator 1210 can generate new converted image data CDT using the new zoom information Info_Z for at least one of the new first image data IDT1 and the new second image data IDT2.

[0063] For example, the new zoom information Info_Z includes a target zoom factor representing a zoom factor of 1.5x and a region of interest representing an upper right region of the image region. In this case, the transformed image generator 1210 may crop the upper right region of the first image data IDT1 corresponding to the zoom factor of 1.5x, generate a cropped region, and scale the generated cropped region to generate transformed image data CDT having a zoom factor of 1.5x. As another example, the received zoom information Info_Z includes a target zoom factor representing a zoom factor of 5.5x and a region of interest representing a lower right region of the image region. In this case, the transformed image generator 1210 may crop the lower right region of the second image data IDT2 corresponding to the zoom factor of 5.5x, generate a cropped region, and scale the generated cropped region to generate transformed image data CDT having a zoom factor of 5.5x.

[0064] The camera module controller 1230 receives zoom information Info_Z from the zoom information generator 1250 and may generate a control signal CTRL for controlling the second camera module 1100b based on a zoom parameter included in the received zoom information Info_Z. For example, the camera module controller 1230 may generate a control signal CTRL for controlling the second camera module 1100b to change the viewing angle of the second camera module 1100b to a viewing angle corresponding to a region of interest, and provide the generated control signal CTRL to the second camera module 1100b via an I2C (Inter Integrated Circuit) bus or the like. Meanwhile, a specific operation of the camera module controller 1230 for generating the control signal CTRL will be described later with reference to FIG. 9.

[0065] In summary, the image processing device 1200 first displays a first image captured by the first camera module 1100a, which is a wide-angle camera, to the user, and can display a first image with an enlarged region of interest using the first image until a certain zoom ratio is reached. At the same time, the image processing device 1200 can pre-adjust the viewing angle of the second camera module 1100b, which is a telephoto camera, to correspond to the region of interest. Then, once a certain zoom ratio is reached (i.e., camera module switching is performed), the image processing device 1200 can display a second image with an enlarged region of interest using the second image captured by the second camera module 1100b.

[0066] In this way, the image processing device 1200 according to the technical concept of the present invention can reduce the delay required to display an image corresponding to a region of interest when switching camera modules due to an increase in zoom magnification by pre-adjusting the viewing angle of the camera module corresponding to a high zoom magnification (i.e., the second camera module 1100b) to correspond to the region of interest. Also, the image processing device 1200 according to the technical concept of the present invention can reduce or prevent the inconvenience of a user having to directly adjust the angle of the camera module so that the region of interest is included in the viewing angle of the camera module corresponding to a high zoom magnification.

[0067] FIG. 5 is a flowchart illustrating a zoom parameter update operation of the zoom information generator 1250 according to an embodiment of the present invention, FIG. 6 is a diagram illustrating touch gestures associated with a zoom function according to an embodiment of the present invention, FIG. 7 is a diagram illustrating a method for calculating a region of interest according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating touch gestures associated with a zoom function according to another embodiment of the present invention.

[0068] 5, the zoom information generator 1250 may receive coordinate information of a user's touch gesture through the input device 1400 (S110). In one example, the zoom information generator 1250 may receive the coordinate information of the touch gesture. Then, the zoom information generator 1250 may determine whether the user's touch gesture is a touch gesture associated with a zoom function based on the received coordinate information.

[0069] Then, the zoom information generator 1250 can calculate a target zoom magnification based on the coordinate information (S120). In one example, if the user's touch gesture is a touch gesture corresponding to zooming in or zooming out (e.g., pinch out, pinch in), the zoom information generator 1250 can calculate a target zoom magnification based on the coordinate information.

[0070] 6 , a user may perform a pinch-out gesture corresponding to zooming in on an image IMG displayed on the display 1300. The zoom information generator 1250 may receive coordinate information of the pinch-out gesture. In one example, the zoom information generator 1250 may acquire coordinate information (X coordinate and Y coordinate) of consecutive touch inputs constituting the pinch-out gesture from the input device 1400, which outputs touch input coordinates at a preset period. For example, the zoom information generator 1250 may acquire coordinate information of first touch inputs A1 and B1 and second touch inputs A2 and B2 constituting the pinch-out gesture. The first touch inputs A1 and B1 are touch inputs at a first time t1, and the second touch inputs A2 and B2 are touch inputs at a second time t2 after the first time t1. The difference between the first time t1 and the second time t2 may correspond to the preset period of the input device 1400.

[0071] The zoom information generator 1250 may calculate a first distance between the first touch inputs A1 and B1 and a second distance between the second touch inputs A2 and B2. The zoom information generator 1250 may calculate a target zoom magnification based on the difference between the first distance and the second distance. The degree of magnification desired by the user, which is indicated by the difference between the first distance and the second distance, may vary depending on the zoom magnification of the image displayed on the display 1300 (i.e., the previous zoom magnification). Therefore, the zoom information generator 1250 may calculate the target zoom magnification using a function that takes the difference between the first distance and the second distance and the previous zoom magnification as variables. Meanwhile, the method by which the zoom information generator 1250 calculates the target zoom magnification using coordinate information is not limited to the above example, and various other methods may be used to calculate the target zoom magnification.

[0072] Then, the zoom information generator 1250 can calculate a region of interest based on the coordinate information (S130). In one example, if the user's touch gesture is a touch gesture for selecting a region of interest (e.g., pinch out, pinch in, touch, tap, double tap, long tap, drag-and-drop, pinch drag, etc.), the zoom information generator 1250 can calculate a region of interest based on the coordinate information.

[0073] 6, the zoom information generator 1250 may calculate the center point using coordinate information of the first touch inputs A1 and B1 or coordinate information of the second touch inputs A2 and B2. For example, the zoom information generator 1250 may obtain the coordinates of the center point by calculating the average value of the X coordinates and the average value of the Y coordinates of the first touch inputs A1 and B1.

[0074] On the other hand, a touch gesture may be input by the user when the image displayed on the display 1300 has already been enlarged or reduced. In this case, if the coordinates of the acquired center point are used as they are, another image area may be enlarged or reduced instead of the image area actually desired by the user. Therefore, the zoom information generator 1250 may correct the coordinates of the acquired center point using information related to the image currently displayed on the display 1300, such as a previous target zoom magnification or a previous region of interest.

[0075] In one example, the zoom information generator 1250 may determine coordinates corresponding to the coordinates of the center point on the entire image area of ​​the first image data IDT1 or the second image data IDT2 based on the previous target zoom magnification and the previous region of interest, and may correct the coordinates of the center point using the determined coordinates. For example, if the previously calculated target zoom magnification falls within a first zoom magnification range (e.g., 1.0x or more and less than 5.0x), the zoom information generator 1250 may determine coordinates corresponding to the coordinates of the center point on the entire image area of ​​the first image data IDT1, and if the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or more), the zoom information generator 1250 may determine coordinates corresponding to the coordinates of the center point on the entire image area of ​​the second image data IDT2.

[0076] The zoom information generator 1250 can then calculate a region of interest based on the coordinates of the corrected center point and the target zoom magnification. Referring to FIG. 7, the zoom information generator 1250 can calculate a region of interest from the resolution of the image data IDT (e.g., the first image data IDT1 or the second image data IDT2). The resolution of the image data IDT used to calculate the region of interest is the resolution of the image data IDT corresponding to the target zoom magnification, among the first image data IDT1 and the second image data IDT2. In a non-limiting example, the zoom information generator 1250 can use the first image data IDT1 if the target zoom magnification falls within a first zoom magnification range (e.g., 1.0x or more and less than 5.0x), and can use the second image data IDT2 if the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or more).

[0077] The zoom information generator 1250 can then calculate Delta_x1, Delta_x2, Delta_y1, and Delta_y2 using the coordinates of the center point according to the following equations.

[0078]

number

[0079] Then, the zoom information generator 1250 can calculate the coordinates (Left, Right, Top, Bottom) of the four sides that make up the crop area (i.e., the area of ​​interest) to be cropped from the image data IDT using the calculated Delta_x1, Delta_x2, Delta_y1, and Delta_y2 according to the following formula:

[0080]

number

[0081] In one example, the region of interest is expressed by the coordinates of four faces constituting the region, or by the coordinates of four vertices constituting the region. Therefore, the zoom information generator 1250 can determine the calculated coordinates of the four faces (left, right, top, bottom) as the region of interest, or calculate the four vertices based on the calculated coordinates of the four faces (left, right, top, bottom) to calculate the region of interest.

[0082] Meanwhile, in the above example, it has been described that the zoom information generator 1250 calculates the region of interest using the image data IDT corresponding to the target zoom magnification, but the technical concept of the present invention is not limited thereto. For example, the zoom information generator 1250 may calculate the region of interest of each of the first image data IDT1 and the second image data IDT2 using coordinates of the center point. In addition, the method of calculating the region of interest and the type of data representing the region of interest are not limited to the above example, and various calculation methods and various data types may be applied.

[0083] Then, the zoom information generator 1250 may update the target zoom magnification and the region of interest based on the calculated target zoom magnification and the calculated region of interest (S140). In one example, the zoom information generator 1250 may update the target zoom magnification and the region of interest using the calculated target zoom magnification and the calculated region of interest, and generate zoom information Info_Z based on the updated target zoom magnification and the updated region of interest.

[0084] 5 to 7 are illustrated and described assuming that the user's touch gesture is a pinch out, but it goes without saying that the technical concept of the present invention can be applied to other touch gestures corresponding to a zoom function. For example, a tap among touch gestures can correspond to selecting a region of interest. When a tap is input, the zoom information generator 1250 can determine the coordinates of one touch input constituting the tap as the center point in the above example and calculate the region of interest using the above method.

[0085] 8, a pinch drag among touch gestures may also correspond to selection of a region of interest. The zoom information generator 1250 may acquire coordinate information of successive touch inputs constituting a pinch drag. For example, the zoom information generator 1250 may acquire coordinate information of a first touch input A1, B1 and a second touch input A2, B2. The zoom information generator 1250 may then calculate a center point using the coordinate information of the first touch input A1, B1 or the coordinate information of the second touch input A2, B2. The zoom information generator 1250 may then calculate a region of interest using the above-described method.

[0086] FIG. 9 is a flowchart illustrating the operation of generating the control signal CTRL of the camera module controller 1230 according to an embodiment of the present invention.

[0087] 9, the camera module controller 1230 may receive a target zoom magnification and a region of interest (S210). Specifically, the camera module controller 1230 may receive zoom information Info_Z including the target zoom magnification and the region of interest from the zoom information generator 1250.

[0088] Then, the camera module controller 1230 may check whether the target zoom magnification is equal to or greater than a preset zoom magnification (S220). Here, the preset zoom magnification may be equal to or smaller than the zoom magnification (e.g., 5.0x) at which the camera module is switched from the first camera module 1100a to the second camera module 1100b.

[0089] If the target zoom magnification is equal to or greater than the preset zoom magnification (YES in S220), the camera module controller 1230 may generate a control signal CTRL for controlling the change of the viewing angle based on the region of interest. For example, the camera module controller 1230 may calculate the coordinates of the center point of the region of interest. Then, the camera module controller 1230 may check the placement information of the prism 1105 of the second camera module 1100b corresponding to the calculated coordinates of the center point. Then, the camera module controller 1230 may generate a control signal CTRL including the checked placement information. However, the present invention is not limited to the above example, and the control signal CTRL may be generated in various ways.

[0090] On the other hand, when the target zoom magnification is relatively low, the converted image data CDT is generated based on the first image data IDT1 of the first camera module 1100a, and therefore, according to an embodiment, the second camera module 1100b may be implemented to be turned off if the target zoom magnification is less than the preset zoom magnification. In this case, the camera module controller 1230 may further generate a control signal CTRL to turn on the second camera module 1100b if the target zoom magnification is equal to or greater than the preset zoom magnification.

[0091] Then, the camera module controller 1230 may output the generated control signal CTRL (S240). For example, the camera module controller 1230 may output the generated control signal CTRL to the second camera module 1100b through an I2C bus connected to the second camera module 1100b. The second camera module 1100b may adjust the position of the prism 1105 based on the position information of the prism 1105 included in the control signal CTRL.

[0092] On the other hand, if the target zoom magnification is less than the preset zoom magnification (NO in S220), the camera module controller 1230 can terminate the operation without generating the control signal CTRL. On the other hand, if the preset zoom magnification is the default zoom magnification of 1.0x, there is no case where the target zoom magnification is less than the preset zoom magnification, so the camera module controller 1230 can continue to generate the control signal CTRL.

[0093] FIG. 10 is a block diagram showing a specific configuration of the transformed image generator 1210 according to an embodiment of the present invention.

[0094] 10, the conversion image generator 1210 includes a first sub-ISP (image signal processor) 1211, a second sub-ISP 1212, and / or a main ISP 1213. In one example, the main ISP 1213 may include a selection module 1214 and / or a post-processing module 1215.

[0095] The first sub-ISP 1211 and the second sub-ISP 1212 may perform image processing on the first image data IDT1 and the second image data IDT2 to generate the first image signal ISIG1 and the second image signal ISIG2. In one example, the first sub-ISP 1211 and the second sub-ISP 1212 may perform an image scaling operation on the first image data IDT1 and the second image data IDT2 to generate the first image signal ISIG1 and the second image signal ISIG2.

[0096] Specifically, the first sub-ISP 1211 and the second sub-ISP 1212 receive zoom information Info_Z from the zoom information generator 1250, use the zoom information Info_Z to crop image areas corresponding to the regions of interest in the first image data IDT1 and the second image data IDT2, generate cropped areas, rescale the cropped areas to have a resolution corresponding to the target zoom magnification, and generate the first image signal ISIG1 and the second image signal ISIG2.

[0097] Meanwhile, the first sub-ISP 1211 and the second sub-ISP 1212 perform cropping and scaling operations using the zoom information Info_Z in different specific embodiments.

[0098] For example, if the target zoom magnification of the zoom information Info_Z falls within a first zoom magnification range (e.g., 1.0x or more and less than 5.0x), the first sub-ISP 1211 may crop an image region corresponding to the region of interest of the zoom information Info_Z from the first image data IDT1 to generate a cropped region, scale the cropped region to have a resolution corresponding to the target zoom magnification, and generate the first image signal ISIG1. If the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or more), the first sub-ISP 1211 may downscale the first image data IDT1 to a preset resolution regardless of the region of interest, and generate the first image signal ISIG1.

[0099] The second sub-ISP 1212 can crop an image region corresponding to the region of interest from the second image data IDT2 to generate a cropped region, regardless of the target zoom magnification of the zoom information Info_Z, and scale the cropped region to have a resolution corresponding to the target zoom magnification to generate the second image signal ISIG2. Alternatively, the second sub-ISP 1212 can be configured to perform the cropping and scaling operations and generate the second image signal ISIG2 only when the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or greater).

[0100] Meanwhile, in an alternative embodiment, when the target zoom magnification is a zoom magnification (e.g., 5.0x) corresponding to the boundary between the first and second zoom magnification ranges, the first sub-ISP 1211 and the second sub-ISP 1212 may generate a first image signal ISIG1 by scaling a region of interest for the first image data IDT1 and generate a second image signal ISIG2 for the second image data IDT2. In this case, the first sub-ISP 1211 and the second sub-ISP 1212 generate image signals representing the same region of interest at the zoom magnification corresponding to the boundary, and the image signals are superimposed by the main ISP 1213, which will be described later.

[0101] In addition to the image scaling operation, the first and second sub-ISPs 1211 and 1212 may perform other image processing operations, such as collecting statistics for auto exposure (AE), auto focus (AF), and auto white balance (AWB), and changing the image format. In a non-limiting example, the first and second sub-ISPs 1211 and 1212 may perform a demosaic operation upon receiving first and second image data IDT1 and IDT2, which are raw images having a Bayer pattern, to generate first and second image signals ISIG1 and ISIG2. Alternatively, depending on the embodiment, the first and second sub-ISPs 1211 and 1212 may convert image data in RGB format to YUV format, or may process raw images having a tetra pattern, in which RGB color pixels are repeatedly arranged in a 2x2 matrix.

[0102] The main ISP 1213 may include a selection module 1214 and a post-processing module 1215. The selection module 1214 may receive a first image signal ISIG1 and a second image signal ISIG2 from the first sub-ISP 1211 and the second sub-ISP 1212, respectively. The selection module 1214 may select at least one of the first image signal ISIG1 and the second image signal ISIG2 based on a target zoom magnification of the zoom information Info_Z, and provide the selected image signal to the post-processing module 1215 as a third image signal ISIG3.

[0103] The post-processing module 1215 may then perform further image processing operations on the received third image signal ISIG3 to generate converted image data CDT. By way of non-limiting example, the post-processing module 1215 may perform various image processing operations on the third image signal ISIG3, such as defective pixel correction, offset correction, lens distortion correction, color gain correction, shading correction, gamma correction, noise reduction, and sharpening. However, depending on the performance of the first camera module 1100a or the second camera module 1100b, some of the post-processing operations may be omitted. For example, if the first camera module 1100a or the second camera module 1100b includes a high-quality image sensor, defective pixel correction or offset correction may be omitted from the post-processing operations.

[0104] In one example, the selection module 1214 may select the first image signal ISIG1 if the target zoom magnification falls within a first zoom magnification range (e.g., greater than or equal to 1.0x and less than 5.0x). The selection module 1214 may then provide the first image signal ISIG1 to the post-processing module 1215 as a third image signal ISIG3. The post-processing module 1215 may perform a post-processing operation on the received first image signal ISIG1 to generate converted image data CDT.

[0105] The selection module 1214 may select the second image signal ISIG2 if the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or greater). The selection module 1214 may provide the second image signal ISIG2 as a third image signal ISIG3 to the post-processing module 1215. The post-processing module 1215 may perform a post-processing operation on the received second image signal ISIG2 to generate converted image data CDT.

[0106] Meanwhile, according to an alternative embodiment, the selection module 1214 may select both the first image signal ISIG1 and the second image signal ISIG2 if the target zoom magnification falls within a second zoom magnification range (e.g., 5.0x or greater). The selection module 1214 may then provide the first image signal ISIG1 and the second image signal ISIG2 to the post-processing module 1215. The post-processing module 1215 may then merge the received first image signal ISIG1 and second image signal ISIG2, perform a post-processing operation, and generate converted image data CDT. In a non-limiting example, the post-processing module 1215 may merge the downscaled first image signal ISIG1 by superimposing it on the bottom left of the image area of ​​the second image signal ISIG2, and perform a post-processing operation to generate converted image data CDT.

[0107] Meanwhile, the selection module 1214 and the post-processing module 1215 may each be implemented by software or hardware, or a combination of software and hardware such as firmware. When the selection module 1214 or the post-processing module 1215 is implemented by software, the respective functions described above are implemented as programmed source code and loaded into a memory provided inside or outside the converted image generator 1210, and the main ISP 1213 executes the software to implement the functions of the selection module 1214 and the post-processing module 1215. When the selection module 1214 or the post-processing module 1215 is implemented by hardware, the selection module 1214 or the post-processing module 1215 may include logic circuits and registers and can perform the respective functions described above based on the register settings.

[0108] Meanwhile, in FIG. 10, the converted image generator 1210 of FIG. 1 is illustrated and described as including the first sub-ISP 1211, the second sub-ISP 1212, and the main ISP 1213, but the converted image generator 1210 may include only at least some of the components of the first sub-ISP 1211, the second sub-ISP 1212, and the main ISP 1213, with the remaining components being implemented as individual components.

[0109] FIG. 11 is a flowchart illustrating the operation of an image processing device 1200 according to one embodiment of the present invention.

[0110] 1 to 11, the image processing device 1200 may receive first image data IDT1 from a first camera module 1100a (S310). Then, the image processing device 1200 may receive second image data IDT2 from a second camera module 1100b (S320). In one example, the first camera module 1100a may have a wide angle, and the second camera module 1100b may have a narrow angle.

[0111] The image processing device 1200 may update the target zoom magnification and the region of interest based on the user input (S330). For example, the image processing device 1200 may calculate the target zoom magnification and the region of interest based on the user input, which is a touch gesture associated with a zoom function, and update the existing target zoom magnification and the region of interest using the calculated target zoom magnification and the region of interest.

[0112] The image processing device 1200 may select image data corresponding to the updated target zoom magnification from among the first image data IDT1 and the second image data IDT2 (S340). Then, the image processing device 1200 may enlarge or reduce an image area corresponding to a region of interest for the selected image data to generate converted image data CDT (S350). Then, the image processing device 1200 may output the converted image data CDT (S360).

[0113] 12A to 12E are sequence diagrams illustrating operations between components of the electronic device 1000 performing a zoom function. Hereinafter, with reference to FIGS. 12A to 12E, a scenario in which the target zoom magnification gradually increases in response to a user input while the electronic device 1000 performs a zoom function will be described. FIG. 12A illustrates a scenario in which the target zoom magnification is equal to or greater than 1.0x and less than 3.0x, FIG. 12B illustrates a scenario in which the target zoom magnification is equal to or greater than 3.0x and less than 5.0x, FIG. 12C illustrates a scenario in which the target zoom magnification is 5.0x, FIG. 12D illustrates a scenario in which the target zoom magnification exceeds 5.0x, and FIG. 12E illustrates another scenario in which the target zoom magnification exceeds 5.0x. Meanwhile, in describing a scenario in which the target zoom magnification gradually increases, the reference zoom magnifications, i.e., 1.0x, 3.0x, and 5.0x, are merely examples, and other zoom magnifications may be used depending on the embodiment.

[0114] 12A, the first camera module 1100a having a wide angle may generate first image data IDT1 (S405). Then, the first camera module 1100a may transmit the first image data IDT1 to the image processing device 1200 (S410). Specifically, the first camera module 1100a may repeatedly generate new first image data IDT1 at a predetermined frame rate and repeatedly transmit the generated first image data IDT1 to the image processing device 1200. Hereinafter, to avoid redundant description, the details of the first camera module 1100a repeatedly generating the first image data IDT1 will be omitted. Meanwhile, in this embodiment, the second camera module 1100b may also be in an off state.

[0115] The image processing device 1200 may generate first converted image data CDT1 based on the first image data IDT1 (S415). Specifically, the image processing device 1200 may generate the first converted image data CDT1 using zoom parameters including a target zoom magnification and a region of interest. In this case, the zoom parameters may have default values, for example, the target zoom magnification may be 1.0x. Then, the image processing device 1200 may transmit the first converted image data CDT1 to the display 1300 (S420). Then, the display 1300 may display an image based on the first converted image data CDT1 (S425).

[0116] The input device 1400 may provide a touch gesture (e.g., a pinch out) input by the user to the image processing device 1200 as a first user input UI1 (S430). In one example, the input device 1400 is also a touch panel coupled to the display 1300. The image processing device 1200 may update the target zoom magnification and the region of interest based on the received first user input UI1 (S435). For example, the updated target zoom magnification may be 2.0x.

[0117] The image processing device 1200 may enlarge an image region corresponding to a region of interest for the next corresponding first image data IDT1 and generate second converted image data CDT2 having a target zoom magnification (e.g., 2.0x) (S440). Then, the image processing device 1200 may transmit the generated second converted image data CDT2 to the display 1300 (S445). Then, the display 1300 may display an image based on the second converted image data CDT2 (S450).

[0118] 12B, the display 1300 may display an image based on the second converted image data CDT2 (S505, S450 in FIG. 12A). The input device 1400 may provide a touch gesture (e.g., pinch out) input by the user to the image processing device 1200 as a second user input UI2 (S510). The image processing device 1200 may update the target zoom magnification and the region of interest based on the received second user input UI2 (S515). For example, the updated target zoom magnification may be 4.0x.

[0119] The image processing device 1200 may transmit a control signal CTRL to the second camera module 1100b having a narrow angle (S520). Specifically, if the updated target zoom magnification exceeds a preset zoom magnification (e.g., 3.0x), the image processing device 1200 may generate a control signal CTRL for controlling the second camera module 1100b and transmit the control signal CTRL. Here, the control signal CTRL may include a command to change the field of view of the second camera module 1100b to a field of view corresponding to the updated region of interest. Meanwhile, according to an embodiment, the control signal CTRL may include a command to turn on the second camera module 1100b. In this case, the second camera module 1100b is turned on in response to the turn-on command (S525).

[0120] The second camera module 1100b may change the viewing angle to correspond to the region of interest by the control signal CTRL (S530). Then, the second camera module 1100b may generate second image data IDT2 (S535). Then, the second camera module 1100b may transmit the second image data IDT2 to the image processing device 1200 (S540). Specifically, the second camera module 1100b may repeatedly generate new second image data IDT2 at a predetermined frame rate and repeatedly transmit the generated second image data IDT2 to the image processing device 1200. Hereinafter, to avoid redundant description, the details of the second camera module 1100b repeatedly generating the second image data IDT2 will be omitted.

[0121] The image processing device 1200 may enlarge an image region corresponding to a region of interest in the first image data IDT1 to generate third converted image data CDT3 having a target zoom magnification (e.g., 4.0x) (S545). Meanwhile, even if the image processing device 1200 receives the second image data IDT2, it does not use it to generate the third converted image data CDT3, but may use it to perform basic settings for image processing operations such as defective pixel detection. The image processing device 1200 may then transmit the generated third converted image data CDT3 to the display 1300 (S550). The display 1300 may then display an image based on the third converted image data CDT3 (S560).

[0122] 12C, the display 1300 may display an image based on the third converted image data CDT3 (S605, S560 in FIG. 12B). The input device 1400 may provide a touch gesture (e.g., pinch out) input by the user to the image processing device 1200 as a third user input UI3 (S610). The image processing device 1200 may update the target zoom magnification and the region of interest based on the received third user input UI3 (S615). For example, the updated target zoom magnification may be 5.0x.

[0123] The image processing device 1200 may transmit a control signal CTRL to the second camera module 1100b having a narrow angle (S620). Here, the control signal CTRL may include an instruction to change the field of view of the second camera module 1100b to a field of view corresponding to the updated region of interest. The second camera module 1100b may change the field of view to a field of view corresponding to the region of interest in response to the control signal CTRL (S625). Then, the second camera module 1100b may generate second image data IDT2 (S630). Then, the second camera module 1100b may transmit the second image data IDT2 to the image processing device 1200 (S635).

[0124] The image processing device 1200 may enlarge an image area corresponding to the region of interest for the first image data IDT1 and generate third image data IDT3 having a target zoom magnification (e.g., 5.0x) (S640). The image processing device 1200 may then generate fourth image data IDT4 corresponding to the region of interest and having the target zoom magnification for the second image data IDT2 (S645). If the base zoom magnification of the second camera module 1100b is 5.0x, the scaling operation for the second image data IDT2 may be omitted.

[0125] The image processing device 1200 may then combine the third image data IDT3 and the fourth image data IDT4 to generate fourth converted image data CDT4 (S650). When the target zoom magnification is changed from less than 5.0x to 5.0x or greater, a natural image transition is required because an image scaled based on the first image data IDT1 is displayed and then an image scaled based on the second image data IDT2 is displayed. Therefore, when the target zoom magnification is 5.0x, the image processing device 1200 may perform image warping, etc. on the third image data IDT3 and the fourth image data IDT4 to generate fourth converted image data CDT4.

[0126] However, the present invention is not limited thereto, and according to an embodiment, the image processing device 1200 may generate the fourth converted image data CDT4 using only the fourth image data IDT4, i.e., the image warping operation of the third image data IDT3 and the fourth image data IDT4 may be omitted.

[0127] The image processing device 1200 may then transmit the generated fourth converted image data CDT4 to the display 1300 (S655). The display 1300 may then display an image based on the fourth converted image data CDT4 (S660).

[0128] 12C, since there is no overlapping zoom factor between the first zoom magnification range (e.g., zoom magnification less than 5.0x) and the second zoom magnification range (e.g., zoom magnification equal to or greater than 5.0x), the above-described operation is performed only at the zoom magnification of 5.0x, which corresponds to the boundary between the ranges, but the present invention is not limited thereto. For example, the first zoom magnification range and the second zoom magnification range may be embodied as overlapping zoom magnification ranges, and the scaled first image data IDT1 and the scaled second image data IDT2 may be overlapped in the overlapping zoom magnification range to generate the fourth converted image data CDT4.

[0129] Next, referring to FIG. 12D, the display 1300 may display an image based on the fourth converted image data CDT4 (S705, S660 in FIG. 12C). The input device 1400 may provide a touch gesture (e.g., pinch out) input by the user to the image processing device 1200 as a fourth user input UI4 (S710). The image processing device 1200 may update the target zoom magnification and the region of interest based on the received fourth user input UI4 (S715). For example, the updated target zoom magnification may be 6.0x.

[0130] The image processing device 1200 may transmit a control signal CTRL to the second camera module 1100b having a narrow angle (S720). Here, the control signal CTRL may include an instruction to change the field of view of the second camera module 1100b to a field of view corresponding to the updated region of interest. The second camera module 1100b may change the field of view to a field of view corresponding to the region of interest in response to the control signal CTRL (S725). Then, the second camera module 1100b may generate second image data IDT2 (S730). Then, the second camera module 1100b may transmit the second image data IDT2 to the image processing device 1200 (S735).

[0131] The image processing device 1200 may enlarge an image region corresponding to the region of interest in the second image data IDT2 to generate fifth converted image data CDT5 having a target zoom magnification (e.g., 6.0x) (S740). The image processing device 1200 may then transmit the generated fifth converted image data CDT5 to the display 1300 (S745). The display 1300 may then display an image based on the fifth converted image data CDT5 (S750).

[0132] Meanwhile, when the target zoom magnification is greater than 5.0x, the electronic device 1000 may generate converted image data in another manner, which will be described with reference to Figure 12E. Meanwhile, steps S805 to S835 of Figure 12E are substantially similar to steps S705 to S735 of Figure 12D, and therefore, a duplicate description will be omitted.

[0133] 12E, the image processing device 1200 may generate fifth image data IDT5 having a preset resolution using the first image data IDT1 (S840). Here, the preset resolution may be set lower than the original resolution of the first image data IDT1. That is, the image processing device 1200 may downscale the first image data IDT1 to the preset resolution to generate the fifth image data IDT5.

[0134] The image processing device 1200 may then enlarge an image area corresponding to the region of interest for the second image data IDT2 to generate sixth image data IDT6 having a target zoom magnification (S845). The image processing device 1200 may then merge the fifth image data IDT5 and the sixth image data IDT6 to generate sixth converted image data CDT6 (S850). Specifically, the image processing device 1200 may superimpose the fifth image data IDT5 on the bottom left of the image area of ​​the sixth image data IDT6 to generate the sixth converted image data CDT6.

[0135] Then, the image processing device 1200 may transmit the generated sixth converted image data CDT6 to the display 1300 (S855). Then, the display 1300 may display an image based on the sixth converted image data CDT6 (S860).

[0136] 12A to 12E (i.e., 1.0x, 3.0x, and 5.0x) are merely examples, and other zoom magnifications may be used depending on the implementation method. Also, each of the embodiments of Figures 12A to 12E may be selectively applied depending on the implementation method of the electronic device 1000.

[0137] 13A and 13B are diagrams illustrating the scaling results of image data according to zoom magnification and the output image of a display. Specifically, based on the embodiment of FIGS. 12A to 12E, FIGS. 13A and 13B are diagrams illustrating the scaling results of the first image data IDT1 of the first camera module 1100a and the second image data IDT2 of the second camera module 1100b according to a target zoom magnification and the output image of a display 1300.

[0138] When the zoom function of the image processing device 1200 is first started, the zoom parameters may have default values, for example, the target zoom factor may be a default value of 1.0x, and the region of interest may be set to a value equal to the entire image area of ​​the first image data IDT1.

[0139] Referring to FIG. 13A, if the zoom function is first started and the target zoom ratio is 1.0x (ZF = 1.0x), the image processing apparatus 1200 can receive the first image data IDT1 from the first camera module 1100a. On the other hand, when the target zoom ratio is less than a preset zoom ratio, for example, less than 3.0x, the second camera module 1100b is turned off to reduce power consumption. The image processing apparatus 1200 uses the first image data IDT1 to generate the first converted image data CDT1. Since the region of interest is the entire image region of the first image data IDT1 with a default value, the post-processing operation can be performed without a scaling operation, and the first converted image data CDT1 can be generated. Then, a touch gesture related to the zoom function is input from the user, and based on the user input, the target zoom ratio and the region of interest can be updated.

[0140] If the target zoom ratio is greater than 1.0x and less than 3.0x (1.0x < ZF < 3.0x), the image processing apparatus 1200 can use the first image data IDT1 to generate the second converted image data CDT2. The image processing apparatus 1200 can scale the region corresponding to the region of interest from the first image data IDT1 to generate the second converted image data CDT2.

[0141] If the target zoom magnification is greater than or equal to 3.0x and less than 5.0x (3.0x≦ZF<5.0x), the image processing device 1200 may generate third converted image data CDT3 using the first image data IDT1. The image processing device 1200 may generate third converted image data CDT3 by scaling a region corresponding to the region of interest from the first image data IDT1. Meanwhile, because the target zoom magnification is a preset zoom magnification, for example, 3.0x or greater, the second camera module 1100b may be turned on and adjusted to have a viewing angle corresponding to the region of interest. The second camera module 1100b may then output second image data IDT2 including the region of interest to the image processing device 1200. However, in this step, the image processing device 1200 may not perform scaling using the second image data IDT2, but may instead perform basic settings for image processing operations.

[0142] Referring to FIG. 13B, if the target zoom magnification is 5.0x (ZF=5.0x), the image processing device 1200 can scale the area corresponding to the region of interest in the first image data IDT1 and scale the area corresponding to the region of interest in the second image data IDT2.

[0143] The image processing device 1200 can then merge the two scaled image data to generate fourth converted image data CDT4. Since the two scaled image data represent the same region of interest, they can be merged in a manner that the entire region is superimposed.

[0144] On the other hand, if the optical zoom magnification of the second camera module 1100b is 5.0x, the scaling operation for the second image data IDT2 can be omitted, and the image processing device 1200 can generate the fourth converted image data CDT4 by merging the second image data IDT2 and the scaled first image data IDT1. Also, depending on the embodiment, the fourth converted image data CDT4 can be generated using only the second image data IDT2.

[0145] If the target zoom factor is 5.0x or greater (ZF≧5.0x), the image processing device 1200 may downscale the first image data IDT1 to a preset resolution and scale a region of the second image data IDT2 corresponding to the region of interest. The image processing device 1200 may then merge the downscaled first image data IDT1 and the scaled second image data IDT2 to generate sixth converted image data CDT6. For example, the image processing device 1200 may merge the downscaled first image data IDT1 by superimposing it on at least a portion of the scaled second image data IDT2. However, the present invention is not limited thereto, and the image processing device 1200 may merge the downscaled first image data IDT1 and the scaled second image data IDT2 by arranging them side by side without superimposing them.

[0146] On the other hand, in yet another example, if the target zoom magnification is 5.0x or greater (ZF≧5.0x), the image processing device 1200 may scale only the second image data IDT2 without using the first image data IDT1 of the first camera module 1100a to generate converted image data.

[0147] FIG. 14A is a diagram illustrating an image displayed on the display 1300 at a high zoom ratio according to an embodiment of the present invention, and FIG. 14B is a diagram illustrating an operation of a user selecting a region of interest at a high zoom ratio.

[0148] In detail, Figure 14A is a diagram showing an example in which the sixth converted image data CDT6 of Figure 13B is displayed on the display 1300, and Figure 14B is a diagram showing the operation of the user selecting a region of interest while the sixth converted image data CDT6 of Figure 13B is being displayed.

[0149] 14A, the sixth converted image data CDT6 includes a first region A1 corresponding to the scaled second image data IDT2 and a second region A2 corresponding to the downscaled first image data IDT1. A user can view the enlarged region of interest (ROI) through the first region A1. The user can view the entire image area through the second region A2. The region of interest (ROI) corresponding to the currently enlarged region of interest is displayed as a rectangle in the second region A2.

[0150] 14B, while the sixth converted image data CDT6 is being displayed, a touch gesture associated with a zoom function is input from the user to the input device 1400. The image processing device 1200 then determines whether the coordinates of the touch gesture are included in the first area A1 or the second area A2, and can apply different methods for the user to select a region of interest depending on the determination result.

[0151] Specifically, the image processing device 1200 may set different touch gestures for selecting a region of interest for each of the first region A1 and the second region A2. When a touch gesture for the first region A1 or the second region A2 is input, the image processing device 1200 may confirm the type of touch gesture based on coordinate information of the touch gesture and determine whether the confirmed type of touch gesture is a touch gesture for selecting a region of interest in the corresponding region. If the confirmed touch gesture is a touch gesture for selecting a region of interest, the image processing device 1200 may calculate the region of interest based on the coordinate information of the touch gesture using the above-described method.

[0152] For example, a first touch gesture TG1, which is a pinch drag, is input into the first region A1. The image processing device 1200 can determine the type of the first touch gesture TG1 based on coordinate information of the first touch gesture TG1. Since the first touch gesture TG1 is a touch gesture (e.g., zoom in, zoom out, pinch drag, etc.) that selects a region of interest set in the first region A1, the image processing device 1200 can calculate the region of interest based on the coordinate information of the first touch gesture TG1.

[0153] Furthermore, a second touch gesture TG2, which is a tap, is input to the second region A2. The image processing device 1200 can determine the type of the second touch gesture TG2 based on the coordinate information of the second touch gesture TG2. Since the second touch gesture TG2 is a touch gesture (e.g., zoom in, zoom out, tap, drag, etc.) that selects a region of interest set in the second region A2, the image processing device 1200 can calculate the region of interest based on the coordinate information of the second touch gesture TG2.

[0154] Meanwhile, a second touch gesture TG2, which is a drag, is input into the second region A2. If the image processing device 1200 determines that the second touch gesture TG2 is a drag, it may further determine whether the drag, which is the second touch gesture TG2, starts from the region ROI represented by a rectangle. If the second touch gesture TG2 is a drag that starts from outside the region ROI represented by a rectangle, the image processing device 1200 may determine that the second touch gesture TG2 is not a touch gesture for selecting a region of interest.

[0155] 15A-15C are block diagrams of electronic devices 2000, 2000a, 2000b including application processors 2200, 2200a, 2200b according to one embodiment of the present invention.

[0156] 15A, electronic device 2000 includes a camera module group 2100, an application processor 2200, a display 2300, an input device 2400, external memory 2500, and / or a PMIC 2600. In one embodiment, application processor 2200 includes an image processor 2210, a memory controller 2220, and / or internal memory 2230.

[0157] The camera module group 2100 includes multiple camera modules 2100a, 2100b, and 2100c. The multiple camera modules 2100a, 2100b, and 2100c may have different viewing angles. In one example, the first camera module 2100a may have the widest viewing angle, the second camera module 2100b may have a medium viewing angle, and the third camera module 2100c may have the narrowest viewing angle.

[0158] The image data generated by the multiple camera modules 2100a, 2100b, and 2100c can be set to correspond to different target zoom magnifications. In a non-limiting example, the image data generated by the first camera module 2100a can be set to correspond to a target zoom magnification of less than 1.0x, the image data generated by the second camera module 2100b can be set to correspond to a target zoom magnification of 1.0x or more but less than 5.0x, and the image data generated by the third camera module 2100c can be set to correspond to a target zoom magnification of 5.0x or more.

[0159] Each of the plurality of camera modules 2100a, 2100b, and 2100c may have the same or similar structure as the first camera module 1100a and the second camera module 1100b described above in Figure 2. In one example, at least one of the plurality of camera modules 2100a, 2100b, and 2100c may be implemented as a foldable camera including a prism, and the viewing angle may be adjusted by adjusting the arrangement of the prism.

[0160] In one example, the multiple camera modules 2100a, 2100b, and 2100c can be arranged so as to be physically separated from one another. That is, instead of the multiple camera modules 2100a, 2100b, and 2100c sharing the sensing area of ​​a single image sensor, an independent image sensor can be arranged inside each of the multiple camera modules 2100a, 2100b, and 2100c.

[0161] Although the drawings show an embodiment in which three camera modules 2100a, 2100b, and 2100c are arranged, the embodiment is not limited thereto. In one example, the camera module group 2100 can be modified to include only two camera modules. Also, in one embodiment, the camera module group 2100 can be modified to include n camera modules (n is a natural number greater than or equal to 4).

[0162] The image processing device 2210 includes a plurality of sub-processors 2211, 2212, 2213, an image generator 2214, a camera module controller 2216, and a zoom information generator 2218. The image processing device 2210 includes a plurality of sub-processors 2211, 2212, 2213 corresponding to the number of the camera modules 2100a, 2100b, 2100c.

[0163] Image data generated from each of the camera modules 2100a, 2100b, and 2100c can be provided to corresponding sub-processors 2211, 2212, and 2213 through separate image signal lines ISLa, ISLb, and ISLc. For example, image data generated from the first camera module 2100a is provided to the sub-processor 2211 through the image signal line ISLa, image data generated from the second camera module 2100b is provided to the sub-processor 2212 through the image signal line ISLb, and image data generated from the third camera module 2100c is provided to the sub-processor 2213 through the image signal line ISLc. Such image data transmission may be performed using, for example, a Camera Serial Interface (CSI) based on a Mobile Industry Processor Interface (MIPI), but the embodiment is not limited thereto.

[0164] Each of the sub-processors 2211, 2212, and 2213 may operate similarly to the first or second sub-processor 1212 or 1213 of FIG. 10. For example, each of the sub-processors 2211, 2212, and 2213 may receive zoom information from the zoom information generator 2218 and perform an image scaling operation on the image data based on the received zoom information. For example, the sub-processor 2211 may scale an image region corresponding to a region of interest in the image data received from the first camera module 2100a if the target zoom magnification is less than 1.0x. The sub-processor 2212 may scale an image region corresponding to a region of interest in the image data received from the second camera module 2100b if the target zoom magnification is equal to or greater than 1.0x and less than 5.0x. The sub-processor 2213 may scale an image region corresponding to a region of interest in the image data received from the third camera module 2100c if the target zoom magnification is equal to or greater than 5.0x.

[0165] Each of the sub-processors 2211, 2212, and 2213 can perform a scaling operation and output the generated image signal to the image generator 2214. The image generator 2214 can operate similarly to the main ISP 1214 of Fig. 10. In one example, the image generator 2214 can select one of the received image signals based on zoom information and generate transformed image data, or can combine two image signals and generate transformed image data.

[0166] In a non-limiting example, if the target zoom magnification is less than 1.0x, the image generator 2214 may generate converted image data based on the image signal received from the sub-processor 2211. If the target zoom magnification is greater than or equal to 1.0x and less than 5.0x, the image generator 2214 may generate converted image data based on the image signal received from the sub-processor 2212. If the target zoom magnification is greater than or equal to 5.0x, the image generator 2214 may generate converted image data based on the image signal received from the sub-processor 2213, or may combine the image signal received from the sub-processor 2212 and the image signal received from the sub-processor 2213 to generate converted image data.

[0167] The camera module controller 2216 can provide control signals to the plurality of camera modules 2100a, 2100b, and 2100c over control lines CSLa, CSLb, and CSLc. The camera module controller 2216 can operate similarly to the camera module controller 1230 of Figure 1. For example, the camera module controller 2216 can generate control signals to control at least one of the plurality of camera modules 2100a, 2100b, and 2100c to change the field of view angle based on a target zoom factor and a region of interest.

[0168] The zoom information generator 2218 can calculate a target zoom factor and a region of interest based on a touch gesture input through the input device 2400. Then, the zoom information generator 2218 can provide zoom information including the zoom factor and the region of interest to the camera module controller 2216, the sub-processors 2211, 2212, 2213, and the image generator 2214. The zoom information generator 2218 can operate similarly to the zoom information generator 1250 of FIG.

[0169] The application processor 2200 stores the received image data in the internal memory 2230 or the external memory 2500 of the application processor 2200, and then reads the image data from the internal memory 2230 or the external memory 2500 and displays an image generated based on the read image data.

[0170] The PMIC 2600 can supply power, e.g., a power supply voltage, to each of the multiple camera modules 2100a, 2100b, and 2100c. For example, under the control of the application processor 2200, the PMIC 2600 can supply a first power to the first camera module 2100a through a power signal line PSLa, a second power to the second camera module 2100b through a power signal line PSLb, and a third power to the third camera module 2100c through a power signal line PSLc.

[0171] The PMIC 2600 generates power and adjusts the power levels for each of the camera modules 2100a, 2100b, and 2100c in response to a power control signal PCON from the application processor 2200. The power control signal PCON may include a power adjustment signal for each operating mode of the camera modules 2100a, 2100b, and 2100c. For example, the operating mode may include a low-power mode, and the power control signal PCON may include information about the camera modules operating in the low-power mode and the power levels to be set. The levels of power provided to each of the camera modules 2100a, 2100b, and 2100c may be the same or different. Furthermore, the power levels can be dynamically changed.

[0172] 15B includes sub-processors 2211a and 2212a, the number of which is less than the number of camera modules 2100a, 2100b, and 2100c, and a multiplexer 2215a. The multiplexer 2215a can receive image data from each of the first camera module 2100a and the third camera module 2100c and output one of the received image data to the sub-processor 2211a. In one example, the multiplexer 2215a can receive a target zoom factor and output image data corresponding to the target zoom factor to the sub-processor 2211a.

[0173] The sub-processor 2212a may receive image data from the second camera module 2100b. Each of the sub-processors 2211a and 2212a may perform a scaling operation and output the generated image signal to the image generator 2214a. In one example, the sub-processor 2211a may scale an image region corresponding to a region of interest in the image data received from the multiplexer 2213a if the target zoom magnification is less than 1.0x or greater than 5.0x. The sub-processor 2212a may scale an image region corresponding to a region of interest in the image data received from the second camera module 2100b if the target zoom magnification is greater than 1.0x and less than 5.0x.

[0174] Each of the sub-processors 2211a and 2212a can output an image signal generated by performing a scaling operation to the image generator 2214. If the target zoom magnification is less than 1.0x or greater than 5.0x, the image generator 2214 can generate converted image data based on the image signal received from the sub-processor 2211a. If the target zoom magnification is greater than 1.0x and less than 5.0x, the image generator 2214 can generate converted image data based on the image signal received from the sub-processor 2212a.

[0175] According to another possible embodiment, the application processor 2200b in Fig. 15C includes sub-processors 2211b, 2212b, and 2213b corresponding to the number of camera modules 2100a, 2100b, and 2100c, and a multiplexer 2215b. The sub-processors 2211b, 2212b, and 2213b in Fig. 15C can operate in the same manner as the sub-processors 2211, 2212, and 2213 in Fig. 15A.

[0176] The multiplexer 2215b can receive image signals from each of the sub-processors 2211b and 2213b and output one of the received image signals to the image generator 2214b. In one example, the multiplexer 2215b can receive a target zoom magnification and output an image signal corresponding to the target zoom magnification to the image generator 2214b.

[0177] If the target zoom magnification is less than 1.0x or greater than or equal to 5.0x, the image generator 2214b can generate transformed image data based on the image signal received from the multiplexer 2215b, and if the target zoom magnification is greater than or equal to 1.0x but less than 5.0x, the image generator 2214b can generate transformed image data based on the image signal received from the sub-processor 2212b.

[0178] 16A to 16C are diagrams illustrating an image capturing operation of the electronic device 2000 according to an embodiment of the present invention. More specifically, FIG. 16A to 16C are diagrams illustrating an image capturing operation of the electronic device 2000 of FIG. 15A.

[0179] Referring to Fig. 16A, the electronic device 2000 includes a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c arranged side by side in a vertical direction. Alternatively, referring to Fig. 16B, the electronic device 2000 includes a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c arranged side by side in a horizontal direction. Alternatively, referring to Fig. 16C, the electronic device 2000 includes a first camera module 2100a and a second camera module 2100b arranged side by side in a horizontal direction, and a third camera module 2100c arranged below the second camera module 2100b.

[0180] The electronic device 2000 can acquire first image data IDT1, second image data IDT2, and third image data IDT3 by capturing an image of a subject using the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c.

[0181] In one example, the electronic device 2000 can acquire first image data IDT1 of an object through a first camera module 2100a having the widest viewing angle, acquire second image data IDT2 of the object through a second camera module 2100b having a medium viewing angle, and acquire third image data IDT3 of the object through a third camera module 2100c having the narrowest viewing angle.

[0182] According to an embodiment of the present invention, at least one of the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c can change the viewing angle by adjusting the arrangement of the prism according to a control signal from the camera module controller 2216. Meanwhile, in the electronic device 2000, the arrangement positions of the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c are not limited to the above example, and may be arranged in various positions.

[0183] FIG. 17 is a block diagram showing a specific configuration of an application processor 3000 according to an embodiment of the present invention.

[0184] Referring to FIG. 17, the application processor 3000 includes a processor 3100, a RAM 3200, an image processing unit 3300, an internal memory 3400, a PMIC 3500, a camera interface 3600, a non-volatile memory interface 3700, an external memory interface 3800, a display interface 3900 and / or an input device interface 3950.

[0185] The processor 3100 can control the overall operation of the application processor 3000. The processor 3100 can be implemented, for example, as a central processing unit (CPU), a microprocessor, etc., and, depending on the embodiment, can also be implemented as a single computing component having two or more independent processors (or cores), i.e., a multi-core processor. The processor 3100 can process or execute programs and / or data stored in the RAM 3200 (or ROM).

[0186] The RAM 3200 may temporarily store programs, data, and / or instructions. Depending on the embodiment, the RAM 3200 may be implemented as a DRAM or an SRAM. The RAM 3200 may temporarily store images input / output via the interfaces 3600, 3700, 3800, 3900, and 3950 or images generated by the image processing device 3300.

[0187] In one example, the application processor 3000 may further include a ROM, which can store persistent programs and / or data. The ROM can be implemented as an erasable programmable ROM (EPROM) or an EEPROM.

[0188] The image processing device 3300 may correspond to the image processing device 1200 of Figure 1 or the image processing devices 2214, 2214a, and 2214b of Figures 15A to 15C. The camera module controller 3410 and the zoom information generator 3430 are implemented as software and loaded into the internal memory 3400. The image processing device 3300 executes the software to implement the functions of the camera module controller 3410 and the zoom information generator 3430. However, the present invention is not limited thereto, and the camera module controller 3410 and the zoom information generator 3430 may also be implemented as hardware or a combination of software and hardware.

[0189] The camera interface 3600 can interface image data input from a camera module 4100 located outside the application processor 3000. The camera module 4100 includes two or more camera modules 4100. The image data received through the camera interface 3600 is provided to the image processing device 3300, or is stored in the non-volatile memory 4200 or the external memory 4300 via the non-volatile memory interface 3700 or the external memory interface 3800.

[0190] The nonvolatile memory interface 3700 can interface data input from or output to the nonvolatile memory 4200. The nonvolatile memory 4200 can be implemented by, for example, a memory card (MMC, eMMC, SD, microSD), etc.

[0191] The display interface 3900 can interface converted image data to be output to the display 4400. The display 4400 can output data for an image or video through a display such as a liquid-crystal display (LCD) or an active matrix organic light emitting diodes (AMOLED). The input device interface 3950 can interface a user input output from the input device 4500. The input device interface 3950 can be implemented by a touch panel.

[0192] One or more of the aforementioned components may include or be embodied by one or more processing circuits. For example, the processing circuit may be hardware including logic circuits, a hardware / software combination such as a processor executing software, or a combination thereof. More specifically, the processing circuit may include, but is not limited to, a CPU, an Arithmetic Logic Unit (ALU), a digital signal processor, a microcomputer, a Field Programmable Gate Array (FPGA), a System-on-Chip (SoC), a programmable logic unit, a microprocessor, an Application-Specific Integrated Circuit (ASIC), etc.

[0193] As mentioned above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used solely for the purpose of explaining the technical idea of ​​the present invention and are not used to limit the meaning or the scope of the present invention as described in the claims. Therefore, a person skilled in the art will understand that various modifications and equivalent embodiments are possible. Therefore, the true technical scope of protection of the present invention should be determined by the technical idea of ​​the claims. [Industrial Applicability]

[0194] The present invention is applicable to, for example, technical fields related to electronic devices. [Explanation of symbols]

[0195] 1000 electronic devices 1100a First camera module 1100b Second camera module 1110a First image sensor 1110b Second image sensor 1200 Image Processing Device 1210 Conversion Image Generator 1230 Camera Module Controller 1250 Zoom Information Generator 1300 display 1400 Input Device 1500 memory CDT conversion image data CTRL control signal IDT1 First image data IDT2 Second image data UI User Input

Claims

1. In an electronic device, The display and a first camera module having a first field of view (FoV) and generating first image data; a second camera module having a second FoV narrower than the first FoV and generating second image data; 1. An application processor, obtaining zoom information including a zoom factor and a region of interest (ROI) determined based on user input; generating a control signal for changing the second FoV based on the ROI; If a second zoom magnification range includes a zoom magnification greater than a zoom magnification included in the first zoom magnification range, and the zoom magnification included in the zoom information is a first zoom magnification having a value between the first zoom magnification range and the second zoom magnification range, scaling a region of the first image data corresponding to the ROI, scaling a region of the second image data corresponding to the ROI, and generating converted image data by superimposing the entire regions of the scaled first image data and the scaled second image data; controlling the display to display the transformed image data; an application processor; Including, The second camera module is An electronic device that is a foldable camera module including a prism, and that changes the second FoV to an FoV corresponding to the ROI by adjusting the position of the prism according to the control signal.

2. The application processor downscaling the first image data if the zoom factor is within the second zoom factor range; 2. The electronic device of claim 1, wherein the transformed image data is generated by juxtaposing the downscaled first image data and the scaled second image data.

3. The application processor downscaling the first image data if the zoom factor is within the second zoom factor range; 2. The electronic device of claim 1, wherein the converted image data is generated by replacing some areas of the scaled second image data with the downscaled first image data.

4. The application processor generating a control signal to turn off the second camera module if the zoom factor is equal to or less than a second zoom factor that is equal to or smaller than the first zoom factor; The electronic device of claim 1 , further comprising: a control signal for turning on the second camera module when the zoom factor exceeds the second zoom factor.

5. The display a touch panel that generates touch input signals for a user's touch gestures; The application processor The electronic device of claim 1 , further comprising: receiving coordinate information about the touch gesture from the display; and calculating the zoom factor and the ROI based on the coordinate information about the touch gesture.

6. The application processor If it is determined that the touch gesture is a touch gesture for changing a zoom magnification based on coordinate information about the touch gesture, the zoom magnification is calculated; The electronic device according to claim 5 , wherein the ROI is calculated when it is determined that the touch gesture is a touch gesture for selecting an ROI based on coordinate information about the touch gesture.

7. 1. A method of operating an electronic device, comprising: receiving first image data captured at a first field of view (FoV) from a first camera module; receiving second image data captured at a second FoV from a second camera module; calculating a zoom factor and a region of interest (ROI) based on a user's touch gesture; generating a control signal to change the second FoV based on the ROI; the second camera module is a folding camera module including a prism, and adjusting the position of the prism according to the control signal to change the second FoV to a FoV corresponding to the ROI; and if a second zoom magnification range includes a zoom magnification greater than a zoom magnification included in the first zoom magnification range, and the calculated zoom magnification is a first zoom magnification having a value between the first zoom magnification range and the second zoom magnification range, scaling a region of the first image data corresponding to the ROI, scaling a region of the second image data corresponding to the ROI, and generating transformed image data by superimposing the entire regions of the scaled first image data and the scaled second image data; controlling a display of the electronic device to display the transformed image data; A method of operation including:

8. The step of generating the transformed image data comprises: downscaling the first image data if the zoom factor is within the second zoom factor range; generating the transformed image data by replacing some areas of the scaled second image data with the downscaled first image data; The method of claim 7, comprising:

9. The step of generating the transformed image data comprises: downscaling the first image data if the zoom factor is within the second zoom factor range; generating the transformed image data by aligning the downscaled first image data and the scaled second image data; The method of claim 7, comprising:

10. The step of calculating the zoom magnification and the ROI includes: calculating a zoom factor when determining that the touch gesture is a touch gesture for changing a zoom factor based on coordinate information about the touch gesture; and calculating the ROI if it is determined based on coordinate information about the touch gesture that the touch gesture is a touch gesture for selecting an ROI.

Images