Application processor, electronic device, and operating method thereof

Abstract

Disclosed is an electronic device including a display, a first camera module having a first field of view (FoV) and generating first image data, a second camera module having a second FoV smaller than the first FoV and generating second image data, an application processor configured to obtain zoom information including a zoom ratio and a region of interest (ROI) determined based on a user's input, generate converted image data by scaling an image region corresponding to the ROI with respect to image data corresponding to the zoom ratio from among the first image data and the second image data, and control the display to display the converted image data.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0106449, filed on August 24, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to an electronic device including an application processor and a method of operating the electronic device, and more specifically, to an application processor, an electronic device including the application processor, and a method of operating the electronic device using multiple camera modules with different fields of view (FoV) to perform zoom functionality. Background Technology

[0004] In recent years, the demand for camera modules, including image sensors, has been increasing, and in particular, the demand for multi-camera systems that include multiple camera modules has been continuously increasing. For example, smartphones equipped with two or more camera modules with different fields of view (FoV) can display images with various zoom ratios by using two or more camera modules.

[0005] In traditional multi-camera systems, digital zoom is performed based on the center point of the frame when zooming. Therefore, if the object the user wants to see is not near the center point of the frame, the user needs to directly adjust the angle or position of the multi-camera system to bring the desired object to the center. For example, when the multi-camera system performs high-ratio zoom, the FoV (FoV) decreases significantly. Therefore, even a slight change in the angle or position of the multi-camera system can cause the FoV to change rapidly, making adjustments difficult for the user. Summary of the Invention

[0006] It provides an application processor, an electronic device including the application processor, and a method for operating the electronic device to enlarge or reduce image data output by multiple camera modules based on a zoom ratio and region of interest (ROI) determined according to user input.

[0007] According to one aspect of an example embodiment, an electronic device includes: a display; a first camera module having a first field of view (FoV) and configured to generate first image data; a second camera module having a second FoV smaller than the first FoV and configured to generate second image data; and an application processor configured to obtain zoom information including a zoom ratio and a region of interest (ROI) determined based on user input, configured to generate converted image data by scaling an image region corresponding to the ROI relative to the image data corresponding to the zoom ratio from the first image data and the second image data, and configured to control the display to display the converted image data.

[0008] According to one aspect of an example embodiment, an application processor includes: a first image signal processor configured to receive first image data capturing a first FoV from a first camera module and configured to generate a first image signal by performing image processing based on the first image data; a second image signal processor configured to receive second image data capturing a second FoV less than the first FoV from a second camera module and configured to generate a second image signal by performing image processing based on the second image data; and a third image signal processor configured to generate converted image data based on at least one of the first image signal and the second image signal, wherein the first image signal processor, the second image signal processor, and the third image signal processor are configured to obtain zoom information including a zoom ratio and ROI determined based on user input. A first image signal processor is configured to generate a first image signal by scaling an image region corresponding to a region of origin (ROI) relative to first image data when the zoom ratio is included in a first zoom ratio range; a second image signal processor is configured to generate a second image signal by scaling an image region corresponding to a region of origin (ROI) relative to second image data when the zoom ratio is included in a second zoom ratio range; and a third image signal processor is configured to select at least one of the first image signal and the second image signal according to the zoom ratio, and is configured to generate converted image data using the selected image signal.

[0009] According to one aspect of an example embodiment, an operating method of an electronic device includes: receiving first image data capturing a first FoV from a first camera module; receiving second image data capturing a second FoV from a second camera module; calculating a zoom ratio and ROI based on a user's touch gesture; scaling an area corresponding to the ROI relative to the first image data when the zoom ratio is included in a first zoom ratio range; scaling the area corresponding to the ROI relative to the second image data when the zoom ratio is included in a second zoom ratio range; and generating converted image data based on at least one of the scaled first image data and the scaled second image data. Attached Figure Description

[0010] From the following detailed description taken in conjunction with the accompanying drawings, exemplary embodiments of the present invention will become clearer, wherein:

[0011] Figure 1 This is a block diagram of an electronic device according to an example embodiment;

[0012] Figure 2 This is a view of the detailed configuration of the camera module according to an example embodiment;

[0013] Figure 3 This is a view illustrating the image capture operation of an electronic device according to an example embodiment;

[0014] Figure 4 This is a view illustrating the operation of an image processing device according to an example embodiment;

[0015] Figure 5 This is a flowchart illustrating the update operation of zoom parameters of a zoom information generator according to an example embodiment;

[0016] Figure 6 This is a view of touch gestures related to the zoom function according to an example embodiment;

[0017] Figure 7 This is a view illustrating a method for calculating the region of interest (ROI) according to an example embodiment;

[0018] Figure 8 This is a view of touch gestures related to the zoom function according to an example embodiment;

[0019] Figure 9 This is a flowchart illustrating the operation of generating control signals for a camera module controller according to an example embodiment;

[0020] Figure 10 This is a block diagram showing the detailed configuration of the transformed image generator according to an example embodiment;

[0021] Figure 11 This is a flowchart illustrating the operation of an image processing device according to an example embodiment;

[0022] Figures 12A to 12E This is a sequence diagram illustrating the operations between components of an electronic device that performs the zoom function;

[0023] Figure 13A and Figure 13B It is a view showing the result of scaling multiple image data according to the zoom ratio of the display and the output image;

[0024] Figure 14A This is a view of an image displayed on a monitor at a high zoom ratio, according to an example embodiment. Figure 14BThis is a view showing the user's selection of the ROI at a high zoom ratio;

[0025] Figures 15A to 15C This is a block diagram of an electronic device including an application processor according to an example embodiment;

[0026] Figures 16A to 16C This is a view illustrating the image capture operation of an electronic device according to an example embodiment; and

[0027] Figure 17 This is a block diagram showing a detailed configuration of the application processor according to an example embodiment. Detailed Implementation

[0028] Figure 1 This is a block diagram of an electronic device 1000 according to an example embodiment.

[0029] refer to Figure 1 The electronic device 1000 may include a first camera module 1100a, a second camera module 1100b, an image processing device 1200, a display 1300, an input device 1400, and / or a memory 1500. For example, the electronic device 1000 may be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. Portable electronic devices may include laptops, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital cameras, digital video cameras, audio equipment, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-book readers, wearable devices, etc.

[0030] Each of the first camera module 1100a and the second camera module 1100b can capture objects (or bodies) outside the electronic device 1000 and generate image data. For example, the first camera module 1100a may include a first image sensor 1110a. Furthermore, the second camera module 1100b may include a second image sensor 1110b. Each of the first image sensor 1110a and the second image sensor 1110b can convert the light signal of the object into an electrical signal using an optical lens (not shown). For this purpose, the first image sensor 1110a and the second image sensor 1110b may include a pixel array in which a plurality of pixels are arranged in two dimensions. For example, one of a plurality of reference colors may be assigned to each of the plurality of pixels. For example, the plurality of reference colors may include RGB (red, green, and blue) or RGBW (red, green, blue, and white). As a non-limiting example, the first image sensor 1110a and the second image sensor 1110b may be implemented using a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) and may be implemented as various types of photoelectric conversion devices.

[0031] The first camera module 1100a can generate first image data IDT1 using the first image sensor 1110a, and the second camera module 1100b can generate second image data IDT2 using the second image sensor 1110b. The first image data IDT1 and the second image data IDT2 can be referred to as image frames and frame data, respectively. Furthermore, each of the first camera module 1100a and the second camera module 1100b can send the generated first image data IDT1 and second image data IDT2 to the image processing device 1200.

[0032] Each of the first camera module 1100a and the second camera module 1100b can generate new first image data IDT1 or second image data IDT2 at a preset time interval, and the frequency corresponding to the period of generating new image data can be called the frame rate. That is, the frame rate can represent the number of new image data generated per unit time.

[0033] The first camera module 1100a and the second camera module 1100b can have different fields of view (FoV). For example, the second field of view (FoV) of the second camera module 1100b can be smaller than the first FoV of the first camera module 1100a. For example, the first camera module 1100a can be a wide-angle camera, and the second camera module 1100b can be a telephoto camera. Therefore, for the same object, the first camera module 1100a can generate wide-angle first image data IDT1, and the second camera module 1100b can generate narrow-angle second image data IDT2.

[0034] For example, the second camera module 1100b can be implemented as a folded camera including a prism 1105, as described in the following reference. Figure 2 The second FoV of the second camera module 1100b can be changed by adjusting the arrangement of the prism 1105.

[0035] The image processing device 1200 can generate converted image data CDT by performing an image processing operation on at least one of the first image data IDT1 and the second image data IDT2 received from the first camera module 1100a and the second camera module 1100b.

[0036] Display 1300 can display various content (e.g., text, images, videos, icons, or symbols) to a user based on converted image data CDT received from image processing device 1200. For example, display 1300 may include liquid crystal display (LCD), light-emitting diode (LED) display, organic light-emitting diode (OLED) display, microelectromechanical system (MEMS) display, or electronic paper display.

[0037] Input device 1400 can transmit instructions or data input from a user or another external device to another component of electronic device 1000, or can output instructions or data received from another component of electronic device 1000 to a user or another external device. In the example, input device 1400 can be implemented as a touch panel and can sense user touch input. For example, as a touch panel, input device 1400 can sense at least one touch input via a user's body part (e.g., a finger) or a touch-only input device. Furthermore, input device 1400 can convert analog signals corresponding to touch input into digital signals and provide the digital signals as user input UI (e.g., coordinate information, such as X and Y coordinates) to image processing device 1200. In the example, when input device 1400 is a touch panel, input device 1400 and display 1300 can be implemented as a single device. In some example embodiments, for ease of description, it is assumed that input device 1400 is implemented as a touch panel. However, the technical ideas of this disclosure are not limited thereto, and it should be understood that the input device 1400 can be implemented as various devices such as a keyboard or a mouse.

[0038] Memory 1500 is a storage area for storing data and can store, for example, an operating system (OS), various programs, and various types of data (e.g., image data). Memory 1500 can be volatile memory, such as dynamic random access memory (DRAM) or static RAM, or non-volatile memory, such as phase-change RAM (PRAM), resistive RAM (ReRAM), or flash memory. In the example, image processing device 1200 can store converted image data CDT in memory 1500. The converted image data CDT stored in memory 1500 can be used later in image processing device 1200, or it can be stored in a separate storage device.

[0039] According to an example embodiment, the image processing device 1200 can provide a zoom function to magnify or reduce an image displayed on the display 1300. In the example, the image processing device 1200 can magnify or reduce the image according to a digital zoom method or an optical zoom method and display the image on the display 1300. For this purpose, the image processing device 1200 may include a conversion image generator 1210, a camera module controller 1230, and / or a zoom information generator 1250.

[0040] The converted image generator 1210 can generate converted image data CDT based on first image data IDT1 and second image data IDT2 received from the first camera module 1100a and the second camera module 1100b, respectively. In the example, the converted image generator 1210 can generate the converted image data CDT by cropping and scaling (enlarging or reducing) the first image data IDT1 and / or the second image data IDT2 based on zoom parameters.

[0041] Zoom parameters can include various parameters related to the zoom function. For example, zoom parameters can include the user's desired target zoom ratio and the region of interest (ROI) that the user wants to zoom in or zoom out of. The ROI can be divided into pixel units within an image region, or it can be divided into regions larger than pixel units. For example, the ROI can be divided into rectangles with a preset aspect ratio, and the preset aspect ratio can correspond to the aspect ratio of the display 1300 (e.g., 16:9 or 2:1). The types of zoom parameters are not limited to the examples above, and zoom parameters can include a variety of parameters.

[0042] In some example embodiments, the converted image generator 1210 may select image data corresponding to the target zoom ratio from the first image data IDT1 and the second image data IDT2. The converted image generator 1210 may generate converted image data CDT by expanding or shrinking the image region corresponding to the ROI relative to the selected image data at the target zoom ratio.

[0043] Camera module controller 1230 can control a first camera module 1100a and a second camera module 1100b. In an example, camera module controller 1230 can generate a control signal CTRL based on zoom parameters to control the second camera module 1100b, which has a smaller FoV than the first camera module 1100a. For example, camera module controller 1230 can generate a control signal CTRL to control the second FoV of the second camera module 1100b to be changed to a FoV corresponding to the ROI included in the zoom parameters. The second camera module 1100b can adjust the arrangement of the prism 1105 according to the received control signal CTRL to have a FoV corresponding to the ROI.

[0044] Figure 1 The illustration shows that the camera module controller 1230 sends the control signal CTRL only to the second camera module 1100b, but the technical concept of this disclosure is not limited thereto. It should be understood that the camera module controller 1230 may also send the control signal CTRL to the first camera module 1100a.

[0045] The zoom information generator 1250 can calculate each zoom parameter based on user input, and update the zoom information including the zoom parameters based on the calculated zoom parameters. In the example, the zoom information generator 1250 can receive user input UI through input device 1400, calculate the target zoom ratio and ROI based on the received user input UI, and update the existing zoom information.

[0046] For example, when the input device 1400 is a touch panel, the user input UI may include touch gestures. In some example embodiments, touch gestures may consist of a single touch input or continuous touch input. For example, touch gestures may include various gestures such as touch, tap, double tap, long tap, drag, drop, flick, swipe, pinch, etc. The input device 1400 may provide the coordinate information of the touch gestures to the zoom information generator 1250. The zoom information generator 1250 may calculate the target zoom ratio desired by the user and the ROI that the user wants to zoom in or out based on the coordinate information of the touch gestures.

[0047] Furthermore, the zoom information generator 1250 can provide zoom information to the conversion image generator 1210 and the camera module controller 1230. The conversion image generator 1210 and the camera module controller 1230 can use the zoom parameters included in the received zoom information to perform their respective functions.

[0048] Each of the camera module controller 1230 and the zoom information generator 1250 can be implemented in software or hardware, or a combination of software and hardware (such as firmware). When the zoom information generator 1250 or the camera module controller 1230 is implemented in software, the zoom information generator 1250 or the camera module controller 1230 can be implemented as source code programmed with each of the functions described above, and can be loaded into memory provided in the image processing device 1200 or stored in read-only memory (ROM). Furthermore, the functions of the zoom information generator 1250 or the camera module controller 1230 can be implemented by software executed by a processor (e.g., a microprocessor) included in the image processing device 1200. When the zoom information generator 1250 or the camera module controller 1230 is implemented in hardware, the zoom information generator 1250 or the camera module controller 1230 can include logic circuitry and registers, and can perform the various functions described above based on register settings.

[0049] Figure 2 This is a view of the detailed configuration of the camera module according to an example embodiment. In the following, reference will be made to... Figure 2The detailed configuration of the second camera module 1100b is described in detail, but the following description can also be applied to the first camera module 1100a according to the example embodiment.

[0050] refer to Figure 2 The second camera module 1100b may include a prism 1105, an optical path folding element (hereinafter referred to as OPFE) 1120, an actuator 1130, an image sensing device 1140 and / or a memory 1150.

[0051] Prism 1105 can alter the path of light L incident from the outside, including the reflective surface 1107 of the light-reflecting material. In an example, prism 1105 can change the path of light L incident in a first direction X to a second direction Y. In some example embodiments, the angle of incidence of light L incident from the first direction X onto the reflective surface 1107 can be the same as the angle of reflection of light L reflected from the reflective surface 1107 in the second direction Y.

[0052] Furthermore, the prism 1105 can rotate the reflective surface 1107 of the light-reflecting material in direction A or rotate the central axis 1106 itself in direction B around the central axis 1106, changing the path of light L incident in the first direction X to the second direction Y. In some example embodiments, the OPFE 1120 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0053] In the example, as shown, the maximum rotation angle of prism 1105 in direction A is 15 degrees or less in the positive (+) A direction, and can be greater than 15 degrees in the negative (-) A direction. However, the example embodiment is not limited to this. In the example, prism 1105 can move within 20 degrees in the positive (+) or negative (-) B direction, or between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees. In the example embodiment, the angle can move at the same angle in the positive (+) or negative (-) B direction, or can move upward to a substantially similar angle within a range of approximately 1 degree. In the example, prism 1105 can move the reflective surface 1107 of the light-reflecting material in a third direction (e.g., the Z direction) parallel to the extension direction of the central axis 1106.

[0054] OPFE 1120 may include, for example, m groups of optical lenses (where m is a positive integer). The m lenses can be moved in a second direction Y to change the optical zoom ratio of the second camera module 1100b. For example, when Z is the basic optical zoom ratio of the second camera module 1100b, the optical zoom ratio of the second camera module 1100b can be changed to 3Z, 4Z, or 5Z or greater when the m optical lenses included in OPFE 1120 are moved.

[0055] Actuator 1130 can move OPFE 1120 or optical lens to a specific and / or desired position. For example, actuator 1130 can adjust the position of optical lens so that image sensor 1110b is located at the focal length of optical lens for accurate sensing. In addition, actuator 1130 can adjust the arrangement of prism 1105.

[0056] In the example embodiment, actuator 1130 can be drawn from... Figure 1 The camera module controller 1230 receives a control signal CTRL for controlling changes in the field of view (FoV) of the second camera module 1100b, and can adjust the arrangement of the prism 1105 according to the received control signal CTRL. For example, the actuator 1130 can rotate the reflective surface 1107 of the prism 1105 in direction A, or move the central axis 1106 of the prism 1105 in direction B, or move the reflective surface 1107 of the prism 1105 in direction Z, according to the control signal CTRL. Therefore, the field of view of the second camera module 1100b can be changed in the vertical, left, and right directions.

[0057] Image sensing device 1140 may include image sensor 1110b, control logic 1142, and / or memory 1144. Image sensor 1106 can use light L provided through an optical lens to sense the image to be sensed. Control logic 1142 can control all operations of second camera module 1100b. For example, control logic 1142 can control the operation of second camera module 1100b according to control signal CTRL provided from camera module controller 1230.

[0058] The memory 1144 can store information required for the operation of the second camera module 1100b, such as calibration data 1146. Calibration data 1146 may include information required for the second camera module 1100b to generate second image data IDT2 using light L provided from an external source. Calibration data 1146 may include, for example, information about the aforementioned rotation angle, information about the focal length, information about the optical axis, etc. When the second camera module 1100b is implemented as a multi-state camera that changes the focal length according to the position of the optical lens, calibration data 1146 may include the focal length value for each position (or state) of the optical lens and information related to autofocus.

[0059] The memory 1150 may store the second image data IDT2 sensed by the second image sensor 1110b. The memory 1150 may be disposed externally to the image sensing device 1140 and may be implemented in a form that is stacked with the sensor chip constituting the image sensing device 1140. In some example embodiments, the memory 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the example embodiments are not limited thereto.

[0060] In the example, Figure 1 The first camera module 1100a may include an actuator 1130, just like the second camera module 1100b. Therefore, each of the first camera module 1100a and the second camera module 1100b may include multiple calibration data 1146 that may be the same as or different from each other depending on the operation of the included actuator 1130.

[0061] In the example, the first camera module 1100a may be a camera module in the form of a folding camera, including the prism 1105 and OPFE 1120 as described above, similar to the second camera module 1100b. Alternatively, the first camera module 1100a may be a camera module in the form of a vertical camera excluding the prism 1105 and OPFE 1120. Alternatively, the first camera module 1100a may be a depth camera having a vertical camera shape and using infrared (IR) to extract depth information. In some example embodiments, the image processing device 1200 may generate a 3D depth image by merging image data provided from a depth camera with image data provided from another camera module (e.g., the second camera module 1100b).

[0062] The shape of the first camera module 1100a is not limited to the examples described above.

[0063] Figure 3 This is a view illustrating the image capture operation of an electronic device 1000 according to an example embodiment.

[0064] refer to Figure 3 The electronic device 1000 can obtain first image data IDT1 and second image data IDT2 by capturing images of an object based on a first camera module 1100a and a second camera module 1100b. In the example, the electronic device 1000 can obtain the first image data IDT1 of the object through the first camera module 1100a, which has a wide angle, and can obtain the second image data IDT2 of the object through the second camera module 1100b, which has a narrow angle. The first image data IDT1 may have a wider image area than the second image data IDT2, and the first image data IDT1 and the second image data IDT2 may have overlapping areas.

[0065] According to an example embodiment, the second camera module 1100b can change the field of view (FoV) by adjusting the arrangement of the prism 1105 according to the control signal CTRL from the camera module controller 1230. When the field of view of the second camera module 1100b is changed, the overlapping area of ​​the first image data IDT1 and the second image data IDT2 can also be changed.

[0066] Figure 4This is a view illustrating the operation of an image processing device 1200 according to an example embodiment. In the example embodiment, Figure 4 It is shown Figure 1 A view of the zoom function operation of the image processing device 1200. Hereinafter, the image processing device 1200 will be described based on the zoom operation of enlarging an image during the zoom function; however, the following description can be equally applied to the reduction operation of reducing an image according to the example embodiment.

[0067] refer to Figure 1 and Figure 4 By capturing images (IMGs) of the same object, the first camera module 1100a can obtain wide-angle first image data IDT1, and the second camera module 1100b can obtain narrow-angle second image data IDT2. The first camera module 1100a and the second camera module 1100b can output the obtained first image data IDT1 and second image data IDT2 to the conversion image generator 1210, respectively.

[0068] The converted image generator 1210 can generate converted image data CDT by performing image magnification on at least one of the first image data IDT1 and the second image data IDT2 using zoom parameters. In some example embodiments, the converted image generator 1210 can obtain zoom parameters by receiving zoom information Info_Z from the zoom information generator 1250.

[0069] In the example, the conversion image generator 1210 can select image data corresponding to the target zoom ratio from the first image data IDT1 and the second image data IDT2. Furthermore, the conversion image generator 1210 can generate converted image data CDT with the target zoom ratio by expanding the image region corresponding to the ROI relative to the selected image data. In the example, the conversion image generator 1210 can generate a cropped region by cropping the image region corresponding to the ROI from the selected image data, and can generate the converted image data CDT with the target zoom ratio by scaling the cropped region.

[0070] The correspondence between the target zoom ratio and the image data can be based on the characteristics (e.g., FoV) of the camera module that generates the image data. For example, a first zoom ratio range including a relatively small target zoom ratio can be set to correspond to first image data IDT1 generated by a first camera module 1100a having a wide angle (e.g., 1.0x or greater and less than 5.0x). Furthermore, a second zoom ratio range including a relatively large target zoom ratio can be set to correspond to second image data IDT2 generated by a second camera module 1100b having a narrow angle (e.g., 5.0x or greater). However, this disclosure is not limited to the above examples, and the first zoom ratio range and the second zoom ratio range can be implemented with overlapping zoom ratio ranges.

[0071] Therefore, the image conversion generator 1210 can generate converted image data CDT in which the ROI is magnified based on the first image data IDT1 generated by the first camera module 1100a in the first zoom ratio range, and can generate converted image data CDT in which the ROI is magnified based on the second image data IDT2 generated by the second camera module 1100b in the second zoom ratio range. That is, the camera module can be converted at the boundary between the first zoom ratio range and the second zoom ratio range.

[0072] In the example, the first camera module 1100a and the second camera module 1100b can obtain new first image data IDT1 and new second image data IDT2 according to a preset frame rate, and can output them to the conversion image generator 1210. Therefore, the conversion image generator 1210 can periodically receive new first image data IDT1 and new second image data IDT2, and can repeat the above operation by using the received new first image data IDT1 and new second image data IDT2.

[0073] The zoom information generator 1250 can generate zoom information Info_Z, which includes zoom parameters such as the target zoom ratio and ROI, and can provide the generated zoom information Info_Z to the conversion image generator 1210 and the camera module controller 1230. Each of the conversion image generator 1210 and the camera module controller 1230 can internally provide a buffer (not shown) for storing the zoom information Info_Z.

[0074] In the example, zoom information generator 1250 can first generate zoom information Info_Z, which includes a target zoom ratio with a default zoom ratio value and a ROI with a default region value. In a non-limiting example, the default zoom ratio can be set to the same value as the default optical zoom ratio (e.g., 1.0x) of the first camera module 1100a. Additionally, the default region can be set to a rectangular region including the center point of the image region and having the aspect ratio of the display 1300. For example, the default region can be set to the same value as the entire image region of the first image data IDT1.

[0075] Therefore, when the zoom function starts, the conversion image generator 1210 can receive zoom information Info_Z with default values, and can generate converted image data CDT based on the zoom parameters with default values ​​included in the received zoom information Info_Z. In the example embodiment, the conversion image generator 1210 can select first image data IDT1 as the image data corresponding to the default zoom ratio. Additionally, the conversion image generator 1210 can generate a cropped region by cropping an image region corresponding to the default region relative to the first image data IDT1, and can generate converted image data CDT with the default zoom ratio by scaling the cropped region. Furthermore, the generated converted image data CDT can be displayed on the display 1300. On the other hand, as in the example above, when the default region is described with the same value as the entire image region of the first image data IDT1, the conversion image generator 1210 can generate converted image data CDT using the first image data IDT1 without performing cropping and scaling.

[0076] On the other hand, unlike the example above, the converted image generator 1210 can be implemented as having zoom parameters with default values ​​pre-stored in an internally provided buffer (not shown). In some example embodiments, the converted image generator 1210 can use the pre-stored zoom parameters to generate converted image data CDT even before zoom information Info_Z has been received from the zoom information generator 1250.

[0077] In the example, zoom information generator 1250 can update zoom parameters based on user input UI received from input device 1400. Furthermore, zoom information generator 1250 can generate new zoom information Info_Z based on the updated zoom parameters and can provide the new zoom information Info_Z to conversion image generator 1210 and camera module controller 1230.

[0078] In an example embodiment, the zoom information generator 1250 can receive user input UI, including coordinate information of a touch gesture, from the input device 1400, while simultaneously displaying, for example, image data CDT generated based on zoom parameters with default values. Furthermore, the zoom information generator 1250 can determine whether the touch gesture is related to the zoom function based on the coordinate information. When the touch gesture is related to the zoom function, the zoom information generator 1250 can update the zoom parameters based on the user input UI.

[0079] Touch gestures related to zoom functionality may include at least one of a touch gesture corresponding to zoom in, a touch gesture corresponding to zoom out, and a touch gesture for selecting an ROI. In a non-limiting example, a touch gesture corresponding to zoom in may be set to pinch-out to increase the distance between fingers when touched with two fingers (or just touching the input device), and a touch gesture corresponding to zoom out may be set to pinch-in to decrease the distance between fingers. Furthermore, touch gestures for selecting an ROI may be set to touch, tap, double tap, long tap, drag, and pinch drag, as well as the aforementioned pinch-out and pinch gestures.

[0080] When examining touch gestures related to zoom functionality, zoom information generator 1250 can calculate the target zoom ratio and / or ROI based on coordinate information. For example, when the touch gesture is an expansion, zoom information generator 1250 can calculate the target zoom ratio and ROI based on coordinate information. As another example, when the touch gesture is a tap, zoom information generator 1250 can calculate the ROI based on coordinate information. See later. Figures 5 to 8 This describes the detailed operation of the zoom information generator 1250, which calculates zoom ratio and ROI.

[0081] When receiving new zoom information Info_Z from zoom information generator 1250, conversion image generator 1210 can generate new converted image data CDT by using the target zoom ratio and ROI included in the new zoom information Info_Z. In the example embodiment, because the first camera module 1100a and the second camera module 1100b output new first image data IDT1 and new second image data IDT2 at a preset period, conversion image generator 1210 can generate new converted image data CDT by using new zoom information Info_Z for at least one of the new first image data IDT1 and the new second image data IDT2.

[0082] For example, the new zoom information Info_Z may include a target zoom ratio indicating a 1.5x zoom ratio and a Region of Interest (ROI) indicating the upper right region of the image area. In some example embodiments, for second image data IDT2 corresponding to a 1.5x zoom ratio, the conversion image generator 1210 can generate a cropped region by cropping the upper right region, and can generate converted image data CDT with a 1.5x zoom ratio by scaling the generated cropped region. As another example, the received zoom information Info_Z may include a target zoom ratio indicating a 5.5x zoom ratio and a Region of Interest (ROI) indicating the lower right region of the image area. In some example embodiments, for second image data IDT2 corresponding to a 5.5x zoom ratio, the conversion image generator 1210 can generate a cropped region by cropping the lower right region, and can generate converted image data CDT with a 5.5x zoom ratio by scaling the generated cropped region.

[0083] The camera module controller 1230 can receive zoom information Info_Z from the zoom information generator 1250, and can generate a control signal CTRL for controlling the second camera module 1100b based on the zoom parameters included in the received zoom information Info_Z. For example, the camera module controller 1230 can generate a control signal CTRL for controlling the FoV of the second camera module 1100b to change to the FoV corresponding to the ROI, and can provide the generated control signal CTRL to the second camera module 1100b via the inter-integrated circuit (I2C) bus. See below for further details. Figure 9 This describes the detailed operation of the camera module controller 1230 in generating the control signal CTRL.

[0084] In summary, the image processing device 1200 can first display a first image captured by the first camera module 1100a, which is a wide-angle camera, to the user, and can use the first image until it reaches a certain zoom ratio to display a first image in which the ROI is magnified. Simultaneously, the image processing device 1200 can pre-adjust the FoV of the second camera module 1100b, which is a telephoto camera, to correspond to the ROI. Furthermore, when the image processing device 1200 reaches a certain zoom ratio (e.g., when performing a camera module conversion), the image processing device 1200 can display a second image in which the ROI is magnified by using the second image captured by the second camera module 1100b.

[0085] Thus, the image processing apparatus 1200 according to this disclosure can reduce the delay required to display the image corresponding to the ROI when the camera module switches according to the increase of zoom ratio by pre-adjusting the FoV of the camera module (e.g., the second camera module 1100b) corresponding to the high zoom ratio to correspond to the ROI. Furthermore, the image processing apparatus 1200 according to this disclosure can reduce or prevent the inconvenience where the user needs to directly adjust the angle of the camera module to ensure that the ROI is included in the FoV of the camera module corresponding to the high zoom ratio.

[0086] Figure 5 This is a flowchart illustrating the zoom parameter update operation of the zoom information generator 1250 according to an example embodiment. Figure 6 This is a view of touch gestures related to the zoom function according to an example embodiment. Figure 7 This is a view illustrating a method for calculating ROI according to an example embodiment, and Figure 8 This is a view of touch gestures related to the zoom function according to an example embodiment.

[0087] refer to Figure 5 In operation S110, the zoom information generator 1250 can receive the coordinate information of the user's touch gesture through the input device 1400. In this example, the zoom information generator 1250 can receive the coordinate information of the touch gesture. Furthermore, the zoom information generator 1250 can determine whether the user's touch gesture is related to the zoom function based on the coordinate information.

[0088] Furthermore, in operation S120, the zoom information generator 1250 can calculate the target zoom ratio based on coordinate information. In the example, when the user's touch gesture is a touch gesture corresponding to zooming in or out (e.g., expanding, pinching), the zoom information generator 1250 can calculate the target zoom ratio based on coordinate information.

[0089] In some example embodiments, references Figure 6The user can perform a zoom-in corresponding to magnification on the image IMG displayed on the display 1300. Furthermore, the zoom information generator 1250 can receive the coordinate information of the zoom gesture. In this example, the zoom information generator 1250 can obtain the coordinate information (X and Y coordinates) of consecutive touch inputs constituting the zoom gesture from the input device 1400, which outputs the coordinates of touch inputs at a preset period. For example, the zoom information generator 1250 can obtain the coordinate information of the first touch inputs A1 and B1 and the second touch inputs A2 and B2 constituting the zoom gesture. The first touch inputs A1 and B1 are touch inputs at a first time t1, and the second touch inputs A2 and B2 are touch inputs at a second time t2 after the first time t1. The difference between the first time t1 and the second time t2 can correspond to a preset period of the input device 1400.

[0090] Furthermore, the zoom information generator 1250 can calculate a first distance between the first touch inputs A1 and B1, and a second distance between the second touch inputs A2 and B2. Additionally, the zoom information generator 1250 can calculate a target zoom ratio based on the difference between the first and second distances. The user's desired magnification, i.e., the difference between the first and second distances, can vary depending on the zoom ratio of the image displayed on the display 1300 (e.g., a previous zoom ratio). Therefore, the zoom information generator 1250 can calculate the target zoom ratio using a function that uses the difference between the first and second distances and the previous zoom ratio as variables. However, the method by which the zoom information generator 1250 calculates the target zoom ratio using coordinate information is not limited to the example described above, and the target zoom ratio can be calculated using various methods.

[0091] Furthermore, during operation S130, the zoom information generator 1250 can calculate the ROI based on coordinate information. In the example, when a user's touch gesture is used to select an ROI (e.g., expand, pinch, touch, tap, double tap, long tap, drag, pinch, etc.), the zoom information generator 1250 can calculate the ROI based on coordinate information.

[0092] In some example embodiments, refer again Figure 6 The zoom information generator 1250 can calculate the center point using the coordinate information of the first touch inputs A1 and B1 or the coordinate information of the second touch inputs A2 and B2. For example, the zoom information generator 1250 can obtain the coordinates of the center point by calculating the average of the X coordinates and the average of the Y coordinates of the first touch inputs A1 and B1.

[0093] When the image displayed on display 1300 has been zoomed in or out, a touch gesture input by the user can be used. In some example embodiments, when the obtained center point coordinates are used as is, an image area other than the image area actually desired by the user may be zoomed in or out. Therefore, zoom information generator 1250 can correct the obtained center point coordinates by using information related to the image currently displayed on display 1300 (e.g., information such as the previous target zoom ratio or the previous ROI).

[0094] In the example, zoom information generator 1250 can check the coordinates corresponding to the coordinates of the center point on the entire image area of ​​the first image data IDT1 or the second image data IDT2 based on the previous target zoom ratio and the previous ROI, and can use the checked coordinates to correct the coordinates of the center point. For example, zoom information generator 1250 can check the coordinates corresponding to the coordinates of the center point on the entire image area of ​​the first image data IDT1 when the previously calculated target zoom ratio is within the first zoom ratio range (e.g., 1.0x or greater and less than 5.0x), and can check the coordinates corresponding to the coordinates of the center point on the entire image area of ​​the second image data IDT2 when the target zoom ratio is within the second zoom ratio range (e.g., 5.0x or greater).

[0095] Furthermore, the zoom information generator 1250 can calculate the ROI based on the corrected center point coordinates and the target zoom ratio. (Reference) Figure 7 The zoom information generator 1250 can calculate the ROI based on the resolution of the image data IDT (e.g., first image data IDT1 or second image data IDT2). In some example embodiments, the resolution of the image data IDT used to calculate the ROI is the resolution of the image data IDT corresponding to the target zoom ratio in the first image data IDT1 and the second image data IDT2. In a non-limiting example, the zoom information generator 1250 can use the first image data IDT1 when the target zoom ratio is within a first zoom ratio range (e.g., 1.0x or greater and less than 5.0x), or it can use the second image data IDT2 when the target zoom ratio is within a second zoom ratio range (e.g., 5.0x or greater).

[0096] Furthermore, the zoom information generator 1250 can calculate Delta_x1, Delta_x2, Delta_y1, and Delta_y2 using the coordinates of the center point according to the following equation.

[0097]

[0098]

[0099]

[0100]

[0101] Wherein, width and height are the resolution width and height of the image data IDT, zoom ratio is the target zoom level, x and y are the corrected x and y coordinates of the center point, and K is a constant corresponding to the image data IDT. The zoom information generator 1250 can use K corresponding to the first image data IDT1 or K corresponding to the second image data IDT2, depending on the calculated target zoom ratio. For example, the zoom information generator 1250 can use K corresponding to the first image data IDT1 when the target zoom ratio is within the first zoom ratio range (e.g., 1.0x or greater and less than 5.0x), or use K corresponding to the second image data IDT2 when the target zoom ratio is within the second zoom ratio range (e.g., 5.0x or greater).

[0102] The zoom information generator 1250 can use the calculated Delta_x1, Delta_x2, Delta_y1, and Delta_y2 to calculate the coordinates of the four surfaces (left, right, top, and bottom) that constitute the cropping region to be cropped from the image data IDT according to the following equation.

[0103] Left=x-Delta_x1

[0104] Right = x + Delta_x2

[0105] Top = y + Delta_y2

[0106] base = y - Delta_y1

[0107] Where x and y are the x and y coordinates of the center point, left and right are the coordinates on the X-axis, and top and bottom are the coordinates on the Y-axis.

[0108] In the example, the ROI can be represented by the coordinates of the four surfaces constituting the region, or by the coordinates of the four vertices constituting the region. Therefore, the zoom information generator 1250 can determine the ROI by the calculated coordinates of the four surfaces (left, right, top, and bottom), or it can calculate the ROI by calculating the four vertices based on the calculated coordinates of the four surfaces (left, right, top, and bottom).

[0109] In the above example, the zoom information generator 1250 has been described as using image data IDT corresponding to the target zoom ratio to calculate the ROI; however, the technical concept of this disclosure is not limited to this. For example, the zoom information generator 1250 can calculate the ROI of each of the first image data IDT1 and the second image data IDT2 by using the coordinates of the center point. Furthermore, the method for calculating the ROI and the data type representing the ROI are not limited to the above examples, and various calculation methods and various types of data can be applied.

[0110] Additionally, in operation S140, the zoom information generator 1250 can update the target zoom ratio and ROI based on the calculated target zoom ratio and the calculated ROI. In the example, the zoom information generator 1250 can use the calculated target zoom ratio and the calculated ROI to update the target zoom ratio and ROI, and can generate zoom information Info_Z based on the updated target zoom ratio and the updated ROI.

[0111] On the other hand, assuming that in the presentation and description Figures 5 to 7 At that time, the user's touch gesture is an expansion. However, it should be understood that the technical ideas of this disclosure can be applied to other touch gestures corresponding to the zoom function. For example, among touch gestures, a tap can correspond to the selection of ROI. When a tap is input, the zoom information generator 1250 can determine the coordinates of a touch input constituting the tap as the center point in the above example, and can calculate the ROI according to the above method.

[0112] As another example, see Figure 8 In touch gestures, pinching and pulling can also correspond to the selection of ROI. The zoom information generator 1250 can obtain the coordinate information of the continuous touch inputs constituting the pinch and pull. For example, the zoom information generator 1250 can obtain the coordinate information of the first touch inputs A1 and B1 and the second touch inputs A2 and B2. Furthermore, the zoom information generator 1250 can calculate the center point using the coordinate information of the first touch inputs A1 and B1 or the second touch inputs A2 and B2. In addition, the zoom information generator 1250 can calculate the ROI according to the above method.

[0113] Figure 9 This is a flowchart illustrating the operation of the control signal CTRL of the camera module controller 1230 according to an example embodiment.

[0114] refer to Figure 9 In operation S210, the camera module controller 1230 may receive the target zoom ratio and ROI. In some example embodiments, the camera module controller 1230 may receive zoom information Info_Z, including the target zoom ratio and ROI, from the zoom information generator 1250.

[0115] Additionally, during operation S220, the camera module controller 1230 can check whether the target zoom ratio is equal to or greater than a preset zoom ratio. The preset zoom ratio can be a value equal to or less than the zoom ratio (e.g., 5.0x) used in the camera module conversion from the first camera module 1100a to the second camera module 1100b.

[0116] When the target zoom ratio is equal to or greater than the preset zoom ratio (as in operation S220), in operation S230, the camera module controller 1230 can generate a control signal CTRL to control the FoV to be changed based on the ROI. For example, the camera module controller 1230 can calculate the coordinates of the center point of the ROI. Furthermore, the camera module controller 1230 can check the arrangement information of the prism 1105 of the second camera module 1100b corresponding to the calculated center point coordinates. Additionally, the camera module controller 1230 can generate a control signal CTRL including the checked arrangement information. However, this disclosure is not limited to the above example, and the control signal CTRL can be generated in various ways.

[0117] When the target zoom ratio is relatively small, converted image data CDT is generated based on the first image data IDT1 of the first camera module 1100a. Therefore, according to the example embodiment, when the target zoom ratio is less than a preset zoom ratio, the second camera module 1100b can be turned off. In some example embodiments, when the target zoom ratio is equal to or greater than the preset zoom ratio, the camera module controller 1230 can additionally generate a control signal CTRL for turning on the second camera module 1100b.

[0118] Additionally, during operation S240, the camera module controller 1230 can output the generated control signal CTRL. For example, the camera module controller 1230 can output the generated control signal CTRL to the second camera module 1100b via an I2C bus connected to the second camera module 1100b. The second camera module 1100b can adjust the arrangement of the prism 1105 based on the arrangement information of the prism 1105 included in the control signal CTRL.

[0119] On the other hand, when the target zoom ratio is less than the preset zoom ratio (No in S220), the camera module controller 1230 can terminate the operation without generating the control signal CTRL. When the preset zoom ratio is 1.0x (default zoom ratio), there is no situation where the target zoom ratio is less than the preset zoom ratio. Therefore, the camera module controller 1230 can continuously generate the control signal CTRL.

[0120] Figure 10 This is a block diagram showing the detailed configuration of the converted image generator 1210 according to an example embodiment.

[0121] refer to Figure 10 The image generator 1210 may include a first sub-image signal processor (ISP) 1211, a second sub-ISP 1212, and / or a main ISP 1213. In this example, the main ISP 1213 may include a selection module 1214 and / or a post-processing module 1215.

[0122] The first sub-ISP 1211 and the second sub-ISP 1212 can generate a first image signal ISIG1 and a second image signal ISIG2 by performing image processing on the first image data IDT1 and the second image data IDT2. In the example, the first sub-ISP 1211 and the second sub-ISP 1212 can generate the first image signal ISIG1 and the second image signal ISIG2 by performing an image scaling operation on the first image data IDT1 and the second image data IDT2.

[0123] In some example embodiments, the first sub-ISP 1211 and the second sub-ISP 1212 can receive zoom information Info_Z from the zoom information generator 1250. They can generate a cropped region by using the zoom information Info_Z to crop the image region corresponding to the ROI of the first image data IDT1 and the second image data IDT2. They can also generate the first image signal ISIG1 and the second image signal ISIG2 by changing the scaling of the cropped region to have a resolution corresponding to the target zoom ratio.

[0124] Each of the example embodiments in the first sub-ISP 1211 and the second sub-ISP 1212 that uses zoom information Info_Z to perform cropping and scaling operations can be different.

[0125] For example, when the target zoom level of zoom information Info_Z is within a first zoom ratio range (e.g., 1.0x or greater and less than 5.0x), the first sub-ISP 1211 can generate a cropped region by cropping an image region corresponding to the ROI of zoom information Info_Z from the first image data IDT1, and can generate a first image signal ISIG1 by scaling the cropped region to have a resolution corresponding to the target zoom ratio. Furthermore, when the target zoom level is within a second zoom ratio range (e.g., 5.0x or greater), the first sub-ISP 1211 can downscale the first image data IDT1 to a preset resolution, regardless of the ROI, to generate the first image signal ISIG1.

[0126] The second sub-ISP 1212 can generate the cropped region by cropping the image region corresponding to the ROI from the second image data IDT2, regardless of the target zoom level of the zoom information Info_Z, and can generate the second image signal ISIG2 by scaling the cropped region to have a resolution corresponding to the target zoom ratio. Alternatively, the second sub-ISP 1212 can be implemented to generate the second image signal ISIG2 by performing cropping and scaling operations only when the target zoom level is within the second zoom ratio range (e.g., 5.0x or greater).

[0127] In other example embodiments, when the target zoom level is a zoom ratio (e.g., 5.0x) corresponding to the boundary between the first zoom ratio range and the second zoom ratio range, the first sub-zoom ratio 1211 and the second sub-zoom ratio 1212 can generate a first image signal ISIG1 by scaling the ROI relative to the first image data IDT1, and can generate a second image signal ISIG2 relative to the second image data IDT2. In some example embodiments, because the first sub-ISP 1211 and the second sub-ISP 1212 generate image signals representing the same ROI at zoom ratios corresponding to the boundary, the first sub-ISP 1211 and the second sub-ISP 1212 can be overlapped with the main ISP 1213, which will be described later.

[0128] In addition to the image scaling operations described above, each of the first sub-ISP 1211 and the second sub-ISP 1212 can perform additional image processing operations, such as collecting statistics on automatic exposure (AE), autofocus (AF), and automatic white balance (AWB), or changing the image format. In a non-limiting example, when receiving first image data IDT1 and second image data IDT2 as original images with a Bayer pattern, the first sub-ISP 1211 and the second sub-ISP 1212 can generate first image signal ISIG1 and second image signal ISIG2 by performing a demosaicing operation. Furthermore, according to an example embodiment, the first sub-ISP 1211 and the second sub-ISP 1212 can convert RGB format image data to YUV format, or can also process original images having a Tetra pattern in which pixels of RGB colors are arranged in 2x2 rows.

[0129] In the example, the main ISP 1213 may include a selection module 1214 and a post-processing module 1215. The selection module 1214 may receive a first image signal ISIG1 and a second image signal ISIG2 from the first sub-ISP 1211 and the second sub-ISP 1212, respectively. Furthermore, the selection module 1214 may select at least one of the first image signal ISIG1 and the second image signal ISIG2 based on the target zoom ratio of the zoom information Info_Z, and may provide the selected image signal as a third image signal ISIG3 to the post-processing module 1215.

[0130] Furthermore, the post-processing module 1215 can generate converted image data CDT by performing additional image processing operations on the received third image signal ISIG3. In a non-limiting example, the post-processing module 1215 can perform various image processing operations on the third image signal ISIG3, such as defect pixel correction, offset correction, lens distortion correction, color gain correction, shading correction, gamma correction, noise reduction, and sharpening. Depending on the performance of the first camera module 1100a or the second camera module 1100b, some of the above post-processing operations may be omitted. For example, when the first camera module 1100a or the second camera module 1100b includes a high-quality image sensor, defect pixel correction or offset correction may be omitted during post-processing operations.

[0131] In the example, when the target zoom ratio is within the first zoom ratio range (e.g., 1.0x or greater and less than 5.0x), the selection module 1214 can select the first image signal ISIG1. Furthermore, the selection module 1214 can provide the first image signal ISIG1 as a third image signal ISIG3 to the post-processing module 1215. The post-processing module 1215 can perform post-processing operations on the received first image signal ISIG1 to generate converted image data CDT.

[0132] Additionally, when the target zoom ratio is within the second zoom ratio range (e.g., 5.0x or greater), the selection module 1214 can select the second image signal ISIG2. Furthermore, the selection module 1214 can provide the second image signal ISIG2 as a third image signal ISIG3 to the post-processing module 1215. The post-processing module 1215 can perform post-processing operations on the received second image signal ISIG2 to generate converted image data CDT.

[0133] However, according to other example embodiments, when the target zoom ratio is within the second zoom ratio range (e.g., 5.0x or greater), the selection module 1214 can select all of the first image signal ISIG1 and the second image signal ISIG2. Furthermore, the selection module 1214 can provide the first image signal ISIG1 and the second image signal ISIG2 to the post-processing module 1215. Additionally, the post-processing module 1215 can merge the received first image signal ISIG1 with the second image signal ISIG2 and perform post-processing operations to generate converted image data CDT. In a non-limiting example, the post-processing module 1215 can merge the reduced first image signal ISIG1 in a manner that overlaps with the lower left of the image region of the second image signal ISIG2, and can perform post-processing operations to generate converted image data CDT.

[0134] Each of the selection module 1214 and the post-processing module 1215 can be implemented in software or hardware, or a combination of software and hardware (such as firmware). When the selection module 1214 or the post-processing module 1215 is implemented in software, each of the above functions can be implemented as programmed source code and loaded into memory provided internally or externally to the conversion image generator 1210. Alternatively, the functions of the selection module 1214 and the post-processing module 1215 can be implemented by the main ISP 1213 executing the software. When the selection module 1214 or the post-processing module 1215 is implemented in hardware, the selection module 1214 or the post-processing module 1215 can include logic circuits and registers, and each of the above functions can be executed based on register settings.

[0135] Assuming in the illustration and description Figure 10 hour, Figure 1 The converted image generator 1210 includes a first sub-ISP 1211, a second sub-ISP 1212, and a main ISP 1213. However, the converted image generator 1210 includes only some of the first sub-ISP 1211, the second sub-ISP 1212, and the main ISP 1213, and the remaining configurations can be implemented as separate configurations.

[0136] Figure 11 This is a flowchart illustrating the operation of an image processing device 1200 according to an example embodiment.

[0137] refer to Figures 1 to 11In operation S310, the image processing device 1200 can receive first image data IDT1 from the first camera module 1100a. Additionally, in operation S320, the image processing device 1200 can receive second image data IDT2 from the second camera module 1100b. In this example, the first camera module 1100a may have a wide-angle lens, and the second camera module 1100b may have a narrow-angle lens.

[0138] In operation S330, the image processing device 1200 can update the target zoom ratio and ROI based on user input. For example, the image processing device 1200 can calculate the target zoom ratio and ROI based on user input as a touch gesture related to the zoom function, and can use the calculated target zoom ratio and ROI to update the existing target zoom ratio and ROI.

[0139] In operation S340, the image processing device 1200 can select image data corresponding to the updated target zoom ratio from the first image data IDT1 and the second image data IDT2. Furthermore, in operation S350, the image processing device 1200 can generate converted image data CDT by expanding or shrinking the image region corresponding to the ROI relative to the selected image data. Finally, in operation S360, the image processing device 1200 can output the converted image data CDT.

[0140] Figures 12A to 12E This is a sequence diagram illustrating the operation between the components of the electronic device 1000 that performs the zoom function. In the following text, reference is made to... Figures 12A to 12E This will describe the situation where the target zoom ratio gradually increases according to the user's input when the electronic device 1000 performs the zoom function. Figure 12A This is a view showing the situation when the target zoom ratio is 1.0x or greater and less than 3.0x. Figure 12B This is a view showing the situation when the target zoom ratio is 3.0x or greater and less than 5.0x. Figure 12C This is a view showing the situation when the target zoom ratio is 5.0x. Figure 12D This is a view used to explain what happens when the target zoom ratio exceeds 5.0x, and Figure 12E This is a view illustrating another scenario where the target zoom ratio exceeds 5.0x. The target zoom ratios used as references in describing the scenario where the target zoom ratio gradually increases—namely, 1.0x, 3.0x, and 5.0x—are merely examples. Other zoom ratios may be used according to the example embodiments.

[0141] refer to Figure 12AIn operation S405, the first camera module 1100a with a wide-angle lens can generate first image data IDT1. Furthermore, in operation S410, the first camera module 1100a can send the first image data IDT1 to the image processing device 1200. In some example embodiments, the first camera module 1100a can repeatedly generate new first image data IDT1 according to a preset frame rate, and can repeatedly send the generated first image data IDT1 to the image processing device 1200. In the following description, to reduce or prevent repetition, the description of the first camera module 1100a repeatedly generating the first image data IDT1 will be omitted. In some example embodiments, the second camera module 1100b can be in a turned-off state.

[0142] In operation S415, image processing device 1200 can generate first transformed image data CDT1 based on first image data IDT1. In some example embodiments, image processing device 1200 can generate first transformed image data CDT1 by using zoom parameters including a target zoom ratio and ROI. In some example embodiments, the zoom parameters can have default values, and for example, the target zoom ratio can be 1.0x. In operation S420, image processing device 1200 can send the first transformed image data CDT1 to display 1300. In operation S425, display 1300 can display an image based on the first transformed image data CDT1.

[0143] In operation S430, input device 1400 may provide touch gesture (e.g., swiping) input from a user to image processing device 1200 as first user input UI1. In the example, input device 1400 may be a touch panel combined with display 1300. In operation S435, image processing device 1200 may update the target zoom ratio and ROI based on the received first user input UI1. For example, the updated target zoom ratio may be 2.0x.

[0144] In operation S440, the image processing device 1200 can generate second-transformed image data CDT2 with a target zoom ratio (e.g., 2.0x) relative to the first image data IDT1 by magnifying the image region corresponding to the ROI. In operation S445, the image processing device 1200 can send the generated second-transformed image data CDT2 to the display 1300. In operation S450, the display 1300 can display an image based on the second-transformed image data CDT2.

[0145] Subsequently, refer to Figure 12B In operating S505 ( Figure 12AIn operation S450, the display 1300 can display an image based on the second converted image data CDT2. In operation S510, the input device 1400 can provide touch gesture (e.g., swipe) input from the user to the image processing device 1200 as a second user input UI2. In operation S515, the image processing device 1200 can update the target zoom ratio and ROI based on the received second user input UI2. For example, the updated target zoom ratio could be 4.0x.

[0146] In operation S520, the image processing device 1200 may send a control signal CTRL to the second camera module 1100b with a narrow angle. In some example embodiments, when the updated target zoom ratio exceeds a preset zoom ratio (e.g., 3.0x), the image processing device 1200 may generate and send a control signal CTRL to control the second camera module 1100b. The control signal CTRL may include an instruction to change the FoV of the second camera module 1100b to the FoV corresponding to the updated ROI. According to example embodiments, the control signal CTRL may include an instruction to turn on the second camera module 1100b. In some example embodiments, in operation S525, the second camera module 1100b may be turned on according to the turn-on instruction.

[0147] In operation S530, the second camera module 1100b can change to the FoV corresponding to the ROI according to the control signal CTRL. Additionally, in operation S535, the second camera module 1100b can generate second image data IDT2. In operation S540, the second camera module 1100b can send the second image data IDT2 to the image processing device 1200. In some example embodiments, the second camera module 1100b can repeatedly generate new second image data IDT2 according to a preset frame rate, and can repeatedly send the generated second image data IDT2 to the image processing device 1200. In the following description, to reduce or prevent repetition, the description of the second camera module 1100b repeatedly generating the second image data IDT2 will be omitted.

[0148] In operation S545, the image processing device 1200 can generate third-transformed image data CDT3 with a target zoom ratio (e.g., 4.0x) relative to the first image data IDT1 by magnifying the image region corresponding to the ROI. Even if the image processing device 1200 receives the second image data IDT2, it is not used to generate the third-transformed image data CDT3, but can be used for basic settings of image processing operations such as defect pixel detection. In operation S550, the image processing device 1200 can send the generated third-transformed image data CDT3 to the display 1300. In operation S560, the display 1300 can display an image based on the third-transformed image data CDT3.

[0149] Subsequently, refer to Figure 12C In operating S605 ( Figure 12B In operation S560, the display 1300 can display an image based on the third-converted image data CDT3. In operation S610, the input device 1400 can provide touch gestures (e.g., swiping) input from the user to the image processing device 1200 as a third user input UI3. In operation S615, the image processing device 1200 can update the target zoom ratio and ROI based on the received third user input UI3. For example, the updated target zoom ratio could be 5.0x.

[0150] In operation S620, the image processing device 1200 can send a control signal CTRL to the second camera module 1100b with a narrow angle. The control signal CTRL can include an instruction to change the FoV of the second camera module 1100b to the FoV corresponding to the updated ROI. In operation S625, the second camera module 1100b can change to the FoV corresponding to the ROI according to the control signal CTRL. Additionally, in operation S630, the second camera module 1100b can generate second image data IDT2. In operation S635, the second camera module 1100b can send the second image data IDT2 to the image processing device 1200.

[0151] In operation S640, the image processing device 1200 can generate third image data IDT3 with a target zoom ratio (e.g., 5.0x) relative to the first image data IDT1 by magnifying the image region corresponding to the ROI. In operation S645, the image processing device 1200 can generate fourth image data IDT4 corresponding to the ROI and having a target zoom ratio relative to the second image data IDT2. When the basic zoom ratio of the second camera module 1100b is 5.0x, the scaling operation on the second image data IDT2 can be omitted.

[0152] In operation S650, the image processing device 1200 can generate fourth converted image data CDT4 by synthesizing the third image data IDT3 and the fourth image data IDT4. When the target zoom ratio changes from less than 5.0x to greater than 5.0x, an image scaled based on the first image data IDT1 is displayed, and then an image scaled based on the second image data IDT2 is displayed; therefore, a natural image conversion is required. Accordingly, when the target zoom ratio is 5.0x, the image processing device 1200 can perform image warping on the third image data IDT3 and the fourth image data IDT4 to generate the fourth converted image data CDT4.

[0153] However, this disclosure is not limited thereto, and according to the example embodiment, the image processing apparatus 1200 can generate the fourth transformed image data CDT4 using only the fourth image data IDT4. That is, the image distortion operations of the third image data IDT3 and the fourth image data IDT4 can be omitted.

[0154] In operation S655, the image processing device 1200 can send the generated fourth-converted image data CDT4 to the display 1300. In operation S660, the display 1300 can display an image based on the fourth-converted image data CDT4.

[0155] Because there are no zoom ratios in which the first zoom ratio range (e.g., zoom ratios less than 5.0x) and the second zoom ratio range (e.g., zoom ratios of 5.0x or higher) overlap with each other, the zoom ratios have been illustrated and described only in relation to... Figure 12C The above operations are performed at a zoom ratio of 5.0x corresponding to the boundaries between the ranges, but this disclosure is not limited thereto. For example, the first zoom ratio range and the second zoom ratio range can be implemented as zoom ratio ranges in which the first zoom ratio range overlaps with the second zoom ratio range, and the fourth transformed image data CDT4 is generated by overlapping the first image data IDT1 and the second image data IDT2 scaled to the zoom ratio range in which the first zoom ratio range overlaps with the second zoom ratio range.

[0156] Subsequently, reference Figure 12D In operating S705 ( Figure 12C In operation S660, display 1300 can display an image based on the fourth converted image data CDT4. In operation S710, input device 1400 can provide touch gesture (e.g., swipe) input from a user to image processing device 1200 as a fourth user input UI4. In operation S715, image processing device 1200 can update the target zoom ratio and ROI based on the received fourth user input UI4. For example, the updated target zoom ratio could be 6.0x.

[0157] In operation S720, the image processing device 1200 can send a control signal CTRL to the second camera module 1100b with a narrow angle. The control signal CTRL can include an instruction to change the FoV of the second camera module 1100b to the FoV corresponding to the updated ROI. In operation S725, the second camera module 1100b can change to the FoV corresponding to the ROI according to the control signal CTRL. Furthermore, in operation S730, the second camera module 1100b can generate second image data IDT2. In operation S735, the second camera module 1100b can send the second image data IDT2 to the image processing device 1200.

[0158] In operation S740, the image processing device 1200 can generate fifth-transformed image data CDT5 with a target zoom ratio (e.g., 6.0x) by magnifying the image region corresponding to the ROI relative to the second image data IDT2. In operation S745, the image processing device 1200 can send the generated fifth-transformed image data CDT5 to the display 1300. In operation S750, the display 1300 can display an image based on the fifth-transformed image data CDT5.

[0159] However, when the target zoom ratio exceeds 5.0x, the electronic device 1000 can generate the converted image data in different ways, as shown in the reference. Figure 12E As stated above. Figure 12E The operation of S805 to S835 can be combined with Figure 12D The operations S705 to S735 are the same or substantially the same, therefore, redundant descriptions will not be given here.

[0160] refer to Figure 12E In operation S840, the image processing device 1200 can generate a fifth image data IDT5 with a preset resolution using the first image data IDT1. The preset resolution can be set to be smaller than the original resolution of the first image data IDT1. That is, the image processing device 1200 can reduce the first image data IDT1 to the preset resolution to generate the fifth image data IDT5.

[0161] In operation S845, the image processing device 1200 can generate a sixth image data IDT6 with a target zoom ratio by magnifying the image region corresponding to the ROI relative to the second image data IDT2. In operation S850, the image processing device 1200 can generate a sixth transformed image data CDT6 by merging the fifth image data IDT5 and the sixth image data IDT6. In some example embodiments, the image processing device 1200 can generate the sixth transformed image data CDT6 by overlaying the fifth image data IDT5 on the lower left of the image region of the sixth image data IDT6.

[0162] In operation S855, the image processing device 1200 can send the generated sixth-converted image data CDT6 to the display 1300. In operation S860, the display 1300 can display an image based on the sixth-converted image data CDT6.

[0163] However, distinction Figures 12A to 12E The zoom ratios (e.g., 1.0x, 3.0x, and 5.0x) in the example embodiments are merely examples, and other zoom ratios can be used depending on the implementation method. Furthermore, depending on the implementation method of the electronic device 1000, different zoom ratios can be selectively applied. Figures 12A to 12E Each of the example embodiments.

[0164] Figure 13A and Figure 13B This is a view showing the result of scaling multiple image data according to the display's zoom ratio and the output image. In some example embodiments, Figure 13A and Figure 13B It shows that based on Figures 12A to 12E The target zoom ratio of the example embodiment is a view of the first image data IDT1 of the first camera module 1100a and the second image data IDT2 of the second camera module 1100b, as well as the output image of the display 1300.

[0165] When the zoom function of the image processing device 1200 is first activated, the zoom parameters can have default values. For example, the target zoom ratio can be 1.0x as the default value, and the ROI can be set to the same value as the entire image area of ​​the first image data IDT1.

[0166] refer to Figure 13AWhen the zoom function is first activated and the target zoom ratio is 1.0x (ZF = 1.0x), the image processing device 1200 can receive first image data IDT1 from the first camera module 1100a. Conversely, when the target zoom ratio is less than a preset zoom ratio, such as 3.0x, the second camera module 1100b can be turned off to reduce power consumption. The image processing device 1200 generates first transformed image data CDT1 using the first image data IDT1. However, since the ROI is the entire image area of ​​the first image data IDT1 as a default value, the image processing device 1200 can generate the first transformed image data CDT1 by performing post-processing operations without scaling. Furthermore, touch gestures related to the zoom function can be input from the user, and the target zoom ratio and ROI can be updated based on the user input.

[0167] When the target zoom ratio exceeds 1.0x and is less than 3.0x (1.0x < ZF < 3.0x), the image processing device 1200 can use the first image data IDT1 to generate the second transformed image data CDT2. The image processing device 1200 can generate the second transformed image data CDT2 by scaling the region corresponding to the ROI in the first image data IDT1.

[0168] When the target zoom ratio is greater than or equal to 3.0x and less than 5.0x (3.0x ≤ ZF < 5.0x), the image processing device 1200 can generate third-transformed image data CDT3 using the first image data IDT1. The image processing device 1200 can generate the third-transformed image data CDT3 by scaling the region corresponding to the ROI in the first image data IDT1. However, since the target zoom ratio is a preset zoom ratio, such as 3.0x or greater, the second camera module 1100b can be turned on and adjusted to have a FoV corresponding to the ROI. Furthermore, the second camera module 1100b can output second image data IDT2, including the ROI, to the image processing device 1200. However, in the current operation, the image processing device 1200 does not perform scaling using the second image data IDT2, but can instead perform basic settings for image processing operations.

[0169] refer to Figure 13B When the target zoom ratio is 5.0x (ZF = 5.0x), the image processing device 1200 can scale the region corresponding to the ROI in the first image data IDT1 and scale the region corresponding to the ROI in the second image data IDT2.

[0170] Furthermore, the image processing device 1200 can generate fourth-transformed image data CDT4 by merging two scaled image data sets. In some example embodiments, because the two scaled image data sets display the same ROI, the entire region can be merged in an overlapping manner.

[0171] When the optical zoom ratio of the second camera module 1100b is 5.0x, the scaling operation on the second image data IDT2 can be omitted, and the image processing device 1200 can generate the fourth transformed image data CDT4 by merging the second image data IDT2 and the scaled first image data IDT1. Furthermore, according to the example embodiment, the image processing device 1200 can generate the fourth transformed image data CDT4 using only the second image data IDT2.

[0172] When the target zoom ratio is 5.0x or higher (ZF ≥ 5.0x), the image processing device 1200 can reduce the first image data IDT1 to a preset resolution and scale the region in the second image data IDT2 corresponding to the ROI. Furthermore, the image processing device 1200 can generate sixth-transformed image data CDT6 by merging the reduced first image data IDT1 and the scaled second image data IDT2. For example, the image processing device 1200 can merge the reduced first image data IDT1 and the scaled second image data IDT2 by overlapping the reduced first image data IDT1 onto at least a portion of the scaled second image data IDT2. However, this disclosure is not limited to this, and the image processing device 1200 can merge the reduced first image data IDT1 and the scaled second image data IDT2 in a manner where they are arranged side-by-side without overlapping.

[0173] On the other hand, in another example, when the target zoom ratio is 5.0x or greater (ZF≥5.0x), the image processing device 1200 can generate converted image data by scaling only the second image data IDT2 without using the first image data IDT1 of the first camera module 1100a.

[0174] Figure 14A This is a view of an image displayed by display 1300 at a high zoom ratio according to an example embodiment, and Figure 14B This is a view showing the operation of the user selecting the ROI at a high zoom ratio.

[0175] In some example embodiments, Figure 14A It is shown that Figure 13B The sixth-converted image data CDT is displayed in an example view on monitor 1300, while Figure 14B It is shown in the display Figure 13BThe sixth transformation of the image data CDT is simultaneously viewed from the user's selected ROI operation view.

[0176] refer to Figure 14A The sixth transformed image data CDT may include a first region A1 corresponding to the scaled second image data IDT2 and a second region corresponding to the scaled first image data IDT1. The user can examine the ROI magnified by the user through the first region A1. Furthermore, the user can reference the entire image region through the second region A2. Additionally, in the second region A2, the ROI corresponding to the currently magnified ROI can be displayed within a rectangle.

[0177] refer to Figure 14B While displaying the image data CDT of the sixth transformation, the input device 1400 can receive touch gestures related to the zoom function from the user. Furthermore, the image processing device 1200 can determine whether the coordinates of the touch gesture are included in the first region A1 or the second region A2, and can apply different methods based on the determined result for the user to select the ROI.

[0178] In some example embodiments, the image processing device 1200 can set different touch gestures for selecting a Region of Interest (ROI) for each of the first region A1 and the second region A2. Furthermore, when a touch gesture is input for either the first region A1 or the second region A2, the image processing device 1200 can check the type of the touch gesture based on its coordinate information and determine whether the checked touch gesture is for selecting an ROI from the corresponding region. When the checked touch gesture is for selecting an ROI, the image processing device 1200 can calculate the ROI based on the coordinate information of the touch gesture according to the method described above.

[0179] For example, a first touch gesture TG1, which is a pinch or drag, can be input into the first region A1. The image processing device 1200 can determine the type of the first touch gesture TG1 based on the coordinate information of the first touch gesture TG1. Because the first touch gesture TG1 is a touch gesture used to select the ROI set for the first region A1 (e.g., zoom in, zoom out, pinch, etc.), the image processing device 1200 can calculate the ROI based on the coordinate information of the first touch gesture TG1.

[0180] Furthermore, a second touch gesture TG2, which is a tap, can be input into the second region A2. The image processing device 1200 can determine the type of the second touch gesture TG2 based on its coordinate information. Since the second touch gesture TG2 is a touch gesture used to select the ROI set for the second region A2 (e.g., zoom in, zoom out, tap, drag, etc.), the image processing device 1200 can calculate the ROI based on the coordinate information of the second touch gesture TG2.

[0181] Furthermore, a second touch gesture TG2, which is a drag, can be input into the second area A2. When it is detected that the second touch gesture TG2 is a drag, the image processing device 1200 can further check whether the drag as the second touch gesture TG2 starts from the ROI indicated by the rectangle. When the second touch gesture TG2 starts from outside the ROI indicated by the rectangle, the image processing device 1200 can determine that the second touch gesture TG2 is not a touch gesture for selecting an ROI.

[0182] Figures 15A to 15C This is a block diagram of electronic devices 2000, 2000a, and 2000b, respectively, including application processors 2200, 2200a, and 2200b according to an example embodiment.

[0183] refer to Figure 15A The electronic device 2000 may include a camera module group 2100, an application processor 2200, a display 2300, an input device 2400, external memory 2500, and / or a PMIC (Power Management Integrated Circuit) 2600. In an example embodiment, the application processor 2200 may include an image processing device 2210, a memory controller 2220, and / or internal memory 2230.

[0184] Camera module group 2100 may include multiple camera modules 2100a, 2100b, and 2100c. These multiple camera modules 2100a, 2100b, and 2100c may have different fields of view (Fov). In the example, the first camera module 2100a may have the widest field of view, the second camera module 2100b may have a middle field of view, and the third camera module 2100c may have the narrowest field of view.

[0185] Furthermore, multiple image data generated by multiple camera modules 2100a, 2100b, and 2100c can be configured to correspond to different target zoom ratios. In a non-limiting example, image data generated by the first camera module 2100a can be configured to correspond to a target zoom ratio less than 1.0x, image data generated by the second camera module 2100b can be configured to correspond to a target zoom ratio of 1.0x or greater but less than 5.0x, and image data generated by the third camera module 2100c can be configured to correspond to a target zoom ratio of 5.0x or greater.

[0186] Furthermore, each of the multiple camera modules 2100a, 2100b, and 2100c can have the same characteristics as described above. Figure 2 The first camera module 1100a and the second camera module 1100b described herein have the same or similar structures. In the example, at least one of the plurality of camera modules 2100a, 2100b and 2100c can be implemented as a folding camera including prisms, and the field of view can be adjusted by adjusting the arrangement of the prisms.

[0187] In the example, each of the multiple camera modules 2100a, 2100b, and 2100c can be physically separated and arranged apart from each other. That is, the sensing area of ​​an image sensor is not divided and used by the multiple camera modules 2100a, 2100b, and 2100c, but an independent image sensor can be arranged inside each of the multiple camera modules 2100a, 2100b, and 2100c.

[0188] Although the accompanying drawings show an example embodiment in which three camera modules 2100a, 2100b, and 2100c are arranged, the example embodiment is not limited thereto. In the example, the camera module group 2100 can be modified and implemented to include only two camera modules. Furthermore, in some example embodiments, the camera module group 2100 can be modified and implemented to include n camera modules (n is a natural number of 4 or greater).

[0189] Image processing device 2210 may include multiple subprocessors 2211, 2212, and 2213, an image generator 2214, a camera module controller 2216, and / or a zoom information generator 2218. Image processing device 2210 may include multiple subprocessors 2211, 2212, and 2213 corresponding to the number of multiple camera modules 2100a, 2100b, and 2100c.

[0190] Image data generated from each of the multiple camera modules 2100a, 2100b, and 2100c can be provided to the corresponding subprocessors 2211, 2212, and 2213 via image signal lines ISLa, ISLb, and ISLc, which are separate from each other. For example, image data generated from the first camera module 2100a can be provided to subprocessor 2211 via image signal line ISLa, image data generated from the second camera module 2100b can be provided to subprocessor 2212 via image signal line ISLb, and image data generated from the third camera module 2100c can be provided to subprocessor 2213 via image signal line ISLc. This image data transmission can be performed using, for example, a Camera Serial Interface (CSI) based on the Mobile Industrial Processor Interface (MIPI), but the example embodiment is not limited thereto.

[0191] Each of the subprocessors 2211, 2212, and 2213 can be similar to Figure 10 The system operates via either a first subprocessor 1211 or a second subprocessor 1212. For example, each of subprocessors 2211, 2212, and 2213 can receive zoom information from zoom information generator 2218 and perform image scaling operations on image data based on the received zoom information. For example, when the target zoom ratio is less than 1.0x, subprocessor 2211 can scale the image region corresponding to the ROI in the image data received from the first camera module 2100a. When the target zoom ratio is greater than or equal to 1.0x and less than or equal to 5.0x, subprocessor 2212 can scale the image region corresponding to the ROI in the image data received from the second camera module 2100b. When the target zoom ratio is 5.0x or greater, subprocessor 2213 can scale the image region corresponding to the ROI in the image data received from the third camera module 2100c.

[0192] Each of the subprocessors 2211, 2212, and 2213 can output the image signal generated by performing a scaling operation to the image generator 2214. The image generator 2214 can be similar to... Figure 10 The main ISP 1213 operates. In the example, the image generator 2214 can generate converted image data by selecting one of the received image signals based on zoom information, or by merging two image signals.

[0193] In a non-limiting example, when the target zoom ratio is less than 1.0x, image generator 2214 can generate converted image data based on the image signal received from subprocessor 2211. Additionally, when the target zoom ratio is 1.0x or greater but less than 5.0x, image generator 2214 can generate converted image data based on the image signal received from subprocessor 2212. Furthermore, when the target zoom ratio is 5.0x or greater, image generator 2214 can generate converted image data based on the image signal received from subprocessor 2213, or it can generate converted image data by combining the image signals received from subprocessor 2212 and subprocessor 2213.

[0194] The camera module controller 2216 can provide control signals to multiple camera modules 2100a, 2100b, and 2100c via control lines CSLa, CSLb, and CSLc. The camera module controller 2216 can be similar to... Figure 1 The camera module controller 1230 is used for operation. For example, the camera module controller 2216 can generate control signals for controlling changes in the FoV of at least one of the multiple camera modules 2100a, 2100b, and 2100c based on the target zoom ratio and ROI.

[0195] The zoom information generator 2218 can calculate the target zoom ratio and ROI based on touch gestures input via the input device 2400. Furthermore, the zoom information generator 2218 can provide zoom information, including the zoom ratio and ROI, to the camera module controller 2216, subprocessors 2211, 2212, and 2213, and the image generator 2214. The zoom information generator 2218 can be similar to... Figure 1 The zoom information generator 1250 is used for operation.

[0196] The application processor 2200 can store the received image data in its internal memory 2230 or external memory 2500, can read the image data from its internal memory 2230 or external memory 2500, and can display an image generated based on the read image data.

[0197] The PMIC 2600 can supply power, such as power supply voltage, to each of the multiple camera modules 2100a, 2100b, and 2100c. For example, under the control of the application processor 2200, the PMIC 2600 can supply first power to the first camera module 2100a via power signal line PSLa, supply second power to the second camera module 2100b via power signal line PSLb, and supply third power to the third camera module 2100c via power signal line PSLc.

[0198] The PMIC 2600 can generate power corresponding to each of the plurality of camera modules 2100a, 2100b, and 2100c in response to a power control signal PCON from the application processor 2200, and can also adjust the power level. The power control signal PCON can include a power adjustment signal for each operating mode of the plurality of camera modules 2100a, 2100b, and 2100c. For example, the operating mode can include a low power mode. In some example embodiments, the power control signal PCON can include information about the camera module operating in low power mode and the set power level. The power levels provided to the plurality of camera modules 2100a, 2100b, and 2100c may be the same or different from each other. Furthermore, the power level can be changed dynamically.

[0199] According to other example embodiments, Figure 15B The application processor 2200a may include fewer subprocessors 2211a and 2212a than the number of camera modules 2100a, 2100b, and 2100c, and / or multiplexer 2215a. Multiplexer 2215a may receive multiple image data from each of the first camera module 2100a and the third camera module 2100c, and may output one of the received image data to subprocessor 2211a. In the example, multiplexer 2215a may receive a target zoom ratio and output image data corresponding to the target zoom ratio to subprocessor 2211a.

[0200] Subprocessor 2212a can receive image data from the second camera module 2100b. Furthermore, each of subprocessors 2211a and 2212a can output the image signal generated by performing a scaling operation to image generator 2214a. In the example, when the target zoom ratio is less than 1.0x or 5.0x or greater, subprocessor 2211a can scale the image region corresponding to the ROI in the image data received from multiplexer 2213a. When the target zoom ratio is 1.0x or greater and less than 5.0x, subprocessor 2212a can scale the image region corresponding to the ROI in the image data received from the second camera module 2100b.

[0201] Each of subprocessors 2211a and 2212a can output the image signal generated by performing a scaling operation to image generator 2214. When the target zoom ratio is less than 1.0x or 5.0x or greater, image generator 2214 can generate converted image data based on the image signal received from subprocessor 2211a. Additionally, when the target zoom ratio is 1.0x or greater and less than 5.0x, image generator 2214 can generate converted image data based on the image signal received from subprocessor 2212a.

[0202] According to another example embodiment, Figure 15C The application processor 2200b may include subprocessors 2211b, 2212b and 2213b corresponding to the number of multiple camera modules 2100a, 2100b and 2100c, and / or multiplexer 2215b. Figure 15C Subprocessors 2211b, 2212b, and 2213b can be coupled with... Figure 15A The subprocessors 2211, 2212 and 2213 operate in the same way.

[0203] Multiplexer 2215b can receive image signals from each of subprocessors 2211b and 2213b, and can output one of the received image signals to image generator 2214b. In the example, multiplexer 2215b can receive a target zoom ratio and output an image signal corresponding to the target zoom ratio to image generator 2214b.

[0204] When the target zoom ratio is less than 1.0x or 5.0x or greater, the image generator 2214b can generate converted image data based on the image signal received from the multiplexer 2215b. Additionally, when the target zoom ratio is 1.0x or greater but less than 5.0x, the image generator 2214b can generate converted image data based on the image signal received from the subprocessor 2212b.

[0205] Figures 16A to 16C This is a view illustrating image capture operations of an electronic device 2000 according to an example embodiment. In some example embodiments, Figures 16A to 16C It is shown Figure 15A A view of the image capture operation of the electronic device 2000.

[0206] refer to Figure 16A The electronic device 2000 may include a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c arranged side-by-side in a vertical direction. Alternatively, refer to Figure 16B The electronic device 2000 may include a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c arranged side-by-side in a horizontal direction. Alternatively, refer to Figure 16C The electronic device 2000 may include a first camera module 2100a and a second camera module 2100b arranged side by side in a horizontal direction, and may also include a third camera module 2100c below the second camera module 2100b.

[0207] Electronic device 2000 can obtain first image data IDT1, second image data IDT2 and third image data IDT3 by capturing images of objects based on first camera module 2100a, second camera module 2100b and third camera module 2100c.

[0208] In the example, electronic device 2000 can obtain first image data IDT1 of an object through a first camera module 2100a with the widest field of view, second image data IDT2 of an object through a second camera module 2100b with an intermediate field of view, and third image data IDT3 of an object through a third camera module 2100c with the narrowest field of view.

[0209] According to the example embodiment, at least one of the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c can change the field of view by adjusting the arrangement of the prisms according to the control signal of the camera module controller 2216. In the electronic device 2000, the arrangement positions of the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c are not limited to the above example, and can be arranged in various positions.

[0210] Figure 17 This is a block diagram showing a detailed configuration of the application processor 3000 according to an example embodiment.

[0211] refer to Figure 17 The application processor 3000 may include a processor 3100, RAM 3200, image processing device 3300, internal memory 3400, PMIC 3500, camera interface 3600, non-volatile memory interface 3700, external memory interface 3800, display interface 3900 and / or input / output interface 3950.

[0212] Processor 3100 can control all operations of application processor 3000. Processor 3100 can be implemented as, for example, a CPU, a microprocessor, etc. According to an example embodiment, processor 3100 can be implemented as a computing component having two or more independent processors (or cores), i.e., a multi-core processor. Processor 3100 can process or execute programs and / or data stored in RAM 3200 (or ROM).

[0213] RAM 3200 can temporarily store programs, data, and / or instructions. According to an example embodiment, RAM 3200 can be implemented as dynamic RAM (DRAM) or static RAM (SRAM). RAM 3200 can be input / output via interfaces 3600, 3700, 3800, 3900, and 3950, or can temporarily store images generated by image processing device 3300.

[0214] In this example, the application processor 3000 may also include ROM. The ROM can store continuously used programs and / or data. The ROM can be implemented as an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM).

[0215] Image processing device 3300 can correspond to Figure 1 Image processing equipment 1200 or Figures 15A to 15C Image processing devices 2214, 2214a, and 2214b are provided. The camera module controller 3410 and zoom information generator 3430 can be implemented in hardware and loaded into internal memory 3400. When the image processing device 3300 executes software, the functions of the camera module controller 3410 and zoom information generator 3430 can be implemented. However, this disclosure is not limited thereto, and the camera module controller 3410 and zoom information generator 3430 can be implemented in hardware or a combination of software and hardware.

[0216] The camera interface 3600 can interface for image data input from a camera module 4100 located outside the application processor 3000. The camera module 4100 may include two or more camera modules 4100. Image data received through the camera interface 3600 is provided to the image processing device 3300, or can be stored in non-volatile memory 4200 or external memory 4300 via non-volatile memory interface 3700 or external memory interface 3800.

[0217] The non-volatile memory interface 3700 can interface to data input from or to the non-volatile memory 4200. The non-volatile memory 4200 can be implemented, for example, as a memory card (MMC, eMMC, SD, or micro SD).

[0218] Display interface 3900 can interface to the converted image data output of display 4400. Display 4400 can output image or video data through a display such as a liquid crystal display (LCD) or an active-matrix organic light-emitting diode (AMOLED). Input / output interface 3950 can be used as a user input / output interface from input device 4500. Input / output interface 3950 can be implemented as a touch panel.

[0219] One or more of the elements disclosed above may include or be implemented in: one or more processing circuits, such as hardware including logic circuits; hardware / software combinations, such as a processor executing software; or combinations thereof. For example, more specifically, the processing circuits may include, but are not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0220] While the concepts of the invention have been specifically shown and described with reference to exemplary embodiments thereof, it should be understood that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. An electronic device, comprising: monitor; A first camera module has a first field of view (FoV) and is configured to generate first image data; The second camera module has a second FoV that is smaller than the first FoV and is configured to generate second image data; as well as The application processor is configured as follows: Obtain zoom information, including the zoom ratio and region of interest (ROI) determined based on user input. When the zoom ratio is included in a first zoom ratio range, converted image data is generated by scaling the region corresponding to the ROI in the first image data. When the zoom ratio is included in a second zoom ratio range that includes zoom ratios larger than the first zoom ratio range, a first image signal is generated by reducing the first image data, a second image signal is generated by scaling the region corresponding to the ROI in the second image data, and the converted image data is generated by merging the first and second image signals. Control the display to show the converted image data. The application processor is configured to scale the region corresponding to the ROI in the first image data and scale the region corresponding to the ROI in the second image data when the zoom ratio is a first zoom ratio that is a value between a first zoom ratio range and a second zoom ratio range, and to generate the transformed image data by merging the scaled first image data and the scaled second image data.

2. The electronic device according to claim 1, wherein, The application processor is configured to generate a control signal that changes the second FoV based on the ROI, and The second camera module is configured to change the second FoV to the FoV corresponding to the ROI based on the control signal.

3. The electronic device according to claim 2, wherein, The second camera module is a folding camera module including prisms and is configured to change the second FoV by adjusting the arrangement of the prisms according to a control signal.

4. The electronic device according to claim 1, wherein, The application processor is configured to generate the transformed image data by replacing a portion of the scaled second image data with reduced first image data.

5. The electronic device according to claim 1, wherein, The application processor is configured to generate a control signal to turn off the second camera module when the zoom ratio is equal to or less than the second zoom ratio, and to generate a control signal to turn on the second camera module when the zoom ratio exceeds the second zoom ratio, wherein the second zoom ratio is equal to or less than the first zoom ratio.

6. The electronic device according to claim 1, wherein, The display includes a touch panel configured to generate touch input signals in response to a user's touch gestures, and The application processor is configured to receive coordinate information about touch gestures from the display and to calculate zoom ratio and ROI based on the coordinate information about touch gestures.

7. The electronic device according to claim 6, wherein, The application processor is configured as follows: When the touch gesture is determined to be a gesture used to change the zoom ratio based on the coordinate information about the touch gesture, the zoom ratio is calculated, and When the touch gesture is determined to be a touch gesture used to select an ROI based on the coordinate information about the touch gesture, the ROI is calculated.

8. An application processor, comprising: A first image signal processor is configured to receive first image data from a first camera module that captures a first field of view (FoV), and to generate a first image signal by performing image processing based on the first image data; The second image signal processor is configured to receive second image data from the second camera module, capturing a second FoV smaller than the first FoV, and to generate a second image signal by performing image processing based on the second image data; as well as A third image signal processor is configured to generate converted image data based on at least one of a first image signal and a second image signal. The first image signal processor, the second image signal processor, and the third image signal processor are configured to obtain zoom information including a zoom ratio and a region of interest (ROI) determined based on user input. The first image signal processor is configured to generate a first image signal by scaling the image region corresponding to the ROI relative to the first image data when the zoom ratio is included in a first zoom ratio range. The second image signal processor is configured to generate a second image signal by scaling the image region corresponding to the ROI relative to the second image data when the zoom ratio is included in the second zoom ratio range. The third image signal processor is configured to select at least one of a first image signal and a second image signal according to the zoom ratio, and use the selected image signal to generate converted image data. The third image signal processor is configured to generate the converted image data based on a first image signal when the zoom ratio is included in a first zoom ratio range, and to generate the converted image data based on a second image signal when the zoom ratio is included in a second zoom ratio range. The first image signal processor is configured to generate a first image signal by reducing the first image data when the zoom ratio is included in the second zoom ratio range, and the third image signal processor is configured to generate the converted image data by merging the first image signal with the second image signal.

9. The application processor of claim 8, further comprising a camera module controller configured to generate control signals for controlling the operation of the first camera module and the second camera module. in, The camera module controller is configured to generate a control signal to control the second camera module to have a second FoV corresponding to the ROI.

10. A method of operating an electronic device, the method comprising: Receive first image data from the first camera module, capturing a first field of view (FoV); Receive second image data from the second camera module, capturing the second FoV; Calculate zoom ratio and region of interest (ROI) based on user touch gestures; When the zoom ratio is included in the first zoom ratio range, the region corresponding to the ROI is scaled relative to the first image data. When the zoom ratio is included in the second zoom ratio range, the region corresponding to the ROI is scaled relative to the second image data. as well as Transformed image data is generated based on at least one of scaled first image data and scaled second image data. The method further includes reducing the first image data when the zoom ratio is included in the second zoom ratio range, and generating the converted image data includes generating the converted image data by merging the reduced first image data with the scaled second image data.

11. The operating method according to claim 10 further includes generating a control signal based on the ROI to change the second FoV of the second camera module.

12. The operating method according to claim 10, wherein, The generation of the transformed image data includes: generating the transformed image data by replacing a portion of the scaled second image data with reduced first image data.

13. The operating method according to claim 10, wherein, The generation of the converted image data includes: when the zoom ratio is included in a first zoom ratio range and a second zoom ratio range, generating the converted image data by overlapping scaled first image data with scaled second image data.

14. The operating method according to claim 10, wherein, The calculation of zoom ratio and ROI includes: When it is determined, based on the coordinate information regarding the touch gesture, that the touch gesture is for changing the zoom ratio, the zoom ratio is calculated; and When the touch gesture is determined to be a touch gesture used to select an ROI based on the coordinate information about the touch gesture, the ROI is calculated.

Images