Camera module, imaging device, and image processing method

Abstract

The present application provides a camera module, an image processing method for the camera module, and an imaging device. The camera module may include: an image sensor, the image sensor including an optical device configured to rotate about at least one of an x-axis, a y-axis, and a z-axis perpendicular to each other in response to a mode signal, and the image sensor being configured to generate a plurality of first images, each first image being generated when the optical device is at a different position; and an image signal processor (ISP) configured to process the plurality of first images, wherein the ISP is further configured to obtain a plurality of pre-stored parameters according to the mode signal, correct the plurality of first images by using the plurality of parameters, and generate a second image by combining the corrected first images.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority based on and claims priority to Korean Patent Application No. 10 - 2020 - 0030382, filed with the Korean Intellectual Property Office on March 11, 2020, and Korean Patent Application No. 10 - 2020 - 0092549, filed with the Korean Intellectual Property Office on July 24, 2020, the entire disclosures of which are incorporated herein by reference. Background Art

[0003] The technical concept of the present disclosure relates to a camera module, and more particularly, to a camera module, an imaging device, and an image processing method for obtaining a high - resolution image by using fixed geometric characteristics.

[0004] As the demand for high - resolution images increases, the pixel size is also decreasing. The decrease in pixel size increases crosstalk between adjacent pixels and decreases the signal - to - noise ratio (SNR) due to a small amount of light. Therefore, the pixel size cannot be continuously decreased.

[0005] In order to generate a high - resolution image by combining a plurality of images generated from a single image sensor, methods such as homography transformation, lens distortion correction, and image warping can be used. However, a large amount of computation is generally required to perform homography transformation, lens distortion correction, and image warping in real time. Therefore, other methods for generating a high - resolution image by combining a plurality of images may be useful. Summary of the Invention

[0006] The present disclosure provides a camera module, an imaging device, and an image processing method for obtaining a high - resolution image by using fixed geometric characteristics.

[0007] To achieve the above object, a camera module according to an aspect of the present disclosure may include: an image sensor including an optical device configured to rotate about at least one of x - axis, y - axis, and z - axis perpendicular to each other in response to a pattern signal, and the image sensor is configured to generate a plurality of first images, each first image being generated when the optical device is at a different position; and an image signal processor (ISP) configured to process the plurality of first images, wherein the ISP is further configured to obtain a plurality of previously stored parameters according to the pattern signal, correct the plurality of first images by using the plurality of parameters, and generate a second image by combining the corrected first images.

[0008] An image processing method according to an aspect of the present disclosure includes generating an image with a large viewing angle based on an image with a small viewing angle. The method may include: generating a plurality of first images by rotating an optical device about at least one of mutually perpendicular x, y, and z axes in response to a mode signal, each first image being generated when the optical device is at a different position, the optical device using a lens or a mirror; correcting the plurality of first images based on a first parameter corresponding to the mode signal among a plurality of prestored parameters; and generating a second image larger than the first images by combining the corrected first images.

[0009] An imaging device according to an aspect of the present disclosure may include: a first image sensor configured to generate a plurality of first images by rotating an optical device along at least one of a pitch, a roll, and a yaw direction based on a first mode signal, the optical device using a lens or a mirror; a second image sensor configured to generate a large viewing angle image having a viewing angle larger than that of the plurality of first images; and an image signal processor (ISP) configured to process the plurality of first images, wherein the ISP corrects the plurality of first images based on a plurality of prestored parameters and generates a second image by combining the corrected first images. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a block diagram showing a camera module according to an exemplary embodiment of the present disclosure;

[0012] Figure 2 is a view showing a prism according to an exemplary embodiment of the present disclosure;

[0013] Figure 3 is a view showing a plurality of images sensed by a prism according to an exemplary embodiment of the present disclosure;

[0014] Figure 4A 、 Figure 4B and Figure 4C is a view showing a method of rotating a prism according to an exemplary embodiment of the present disclosure;

[0015] Figure 5 is a view showing a method of generating an image in an image generator according to an exemplary embodiment of the present disclosure;

[0016] Figure 6 is a block diagram showing a post-processing operation in an image signal processor according to an exemplary embodiment of the present disclosure;

[0017] Figure 7 is a flowchart showing an image processing method according to an exemplary embodiment of the present disclosure;

[0018] Figure 8 is a detailed flowchart showing a correction method in an image processing method according to an exemplary embodiment of the present disclosure;

[0019] Figure 9A and Figure 9B is a block diagram showing an imaging device according to an exemplary embodiment of the present disclosure; and

[0020] Figure 10A and Figure 10B is a block diagram showing an imaging device including a plurality of camera modules according to an exemplary embodiment of the present disclosure. Detailed Description of the Invention

[0021] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0022] Figure 1 is a block diagram showing a camera module 10 according to an exemplary embodiment of the present disclosure.

[0023] Referring to Figure 1 , the camera module 10 may include an image sensor 100, an image signal processor (ISP) 200, and a memory 300. The image sensor 100 may include an optical device 110, the ISP 200 may include an image generator 210, and the memory 300 may store parameter data 310. Items described herein as modules, devices, or units or using suffixes such as “…er” or “…body” refer to a combination of hardware, firmware, and / or software, which may include various circuits and other components configured to operate various steps and methods described herein.

[0024] The camera module 10 may capture and / or store an image of an object using a solid-state image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and may be implemented as part of a digital camera, a digital video camera, a mobile phone, a tablet computer, or other portable electronic devices. The portable electronic devices may include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video camera, an audio device, a portable multimedia player (PMP), and a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, etc. In addition, the camera module 10 may be installed on electronic devices such as drones and advanced driver assistance systems (ADAS), or may be installed on electronic devices provided as components in vehicles, furniture, manufacturing facilities, doors, and various measurement devices.

[0025] The image sensor 100 may convert an optical signal of an object incident through an optical lens into an electrical signal, generate image data IDTA based on the electrical signal, and output the image data IDTA and an index IDX corresponding to the sensing order of the image data IDTA. The index IDX may include spatial coordinates of a plurality of image data sets IDTA. The image sensor 100 may be mounted on an electronic device having an image or light sensing function.

[0026] According to an exemplary embodiment of the present disclosure, the image sensor 100 may include an optical device 110. The optical device 110 may be a condensing device including a mirror and a lens. The optical device 110 may obtain a plurality of images for one scene by rotating along at least one of the x-axis, y-axis, and z-axis perpendicular to each other. In the exemplary embodiment, the optical device 110 may obtain different scenes by moving vertically and horizontally by rotating in the pitch and roll directions. The optical device 110 may include or be connected to one or more actuators that receive instructions from a processor to move the light receiving portion of the optical device 110 vertically and horizontally. Accordingly, the optical device 110 or a part of the optical device may move within the camera module 10 relative to the camera module 10.

[0027] Hereinafter, for ease of explanation, as an embodiment of the optical device 110, a lens such as a prism is taken as an example. However, the present invention is not limited thereto, and various optical devices capable of aggregating light reflected by an object by using optical characteristics such as scattering or refraction of light or changing a moving path of light may be used.

[0028] According to an exemplary embodiment of the present disclosure, the image sensor 100 may sense a plurality of image data sets IDTA while moving the optical device 110 vertically and horizontally (i.e., up, down, left, and right) based on a mode signal MODE_S. For example, the image sensor 100 of the present invention may arrange the sensed image data IDTA for one scene in the form of an M×N matrix (M is the number of rows, N is the number of columns, and M and N are integers of 2 or greater) based on the mode signal MODE_S. The optical device 110 may have vertical and horizontal rotation angles restricted within a specific range and rotate within the specific angles to capture images. Accordingly, the image data IDTA sensed by the optical device 110 may have fixed geometric characteristics. As discussed herein, the image data IDTA may refer to the entire image data of an image, and the image data set IDTA (or a set of image data IDTA) may refer to a subset of the data obtained, for example, at different positions of the optical device (e.g., a lens) at different times. A plurality of image data sets IDTA may be obtained and combined and / or processed to obtain an image including the image data IDTA.

[0029] The mode signal MODE_S may indicate a method of arranging the image data IDTA sensed by the image sensor 100. For example, the image sensor 100 may arrange the sensed image data IDTA in a 3×3 matrix based on the mode signal MODE_S indicating the first mode, may arrange the image data IDTA in a 5×5 matrix based on the mode signal MODE_S indicating the second mode, and may arrange the image data IDTA in a 3×4 matrix (or, in some cases, a 4×3 matrix) based on the mode signal MODE_S indicating the third mode. In addition to the method of arranging the image data IDTA, the mode signal MODE_S may indicate a sensing method of obtaining the image data IDTA. According to an exemplary embodiment, the mode signal MODE_S may be set by a user's manipulation. The image sensor 100 or the camera module 10 may include a set of instructions (e.g., computer program code) that cause the optical device 110 to sequentially move to a series of consecutive positions for capturing the image data.

[0030] The image sensor 100 may provide an index IDX indicating the sensing order of the arranged image data IDTA. The index IDX may correspond to the time series of sensing and may be related to the position information or coordinates of the image data IDTA for image correction to be described later.

[0031] Since the sensed image data IDTA has fixed geometric characteristics, the array (and / or number) of the image data set (IDTA) and the index (IDX) may sufficiently include the geometric characteristic information of the image sensor 100 (particularly, the condensing optical device 110) that senses the image data IDTA.

[0032] In Figure 1 , for ease of description, only one image sensor 100 is shown, but the present invention is not limited thereto, and a case where there are multiple image sensors may be assumed. Each of the multiple image sensors may have the same resolution, but the first image sensor may have a relatively narrow viewing angle, and the second image sensor may have a relatively wide viewing angle. The first image sensor may be referred to as a tele sensor, and the second image sensor may be referred to as a wide sensor. A case of using multiple image sensors (or multiple camera modules) will be described later with reference to Figures 10A to 10B Description is made of the case of using multiple image sensors (or multiple camera modules).

[0033] The ISP 200 can receive image data IDTA, which is the output signal of the output buffer of an image sensor, and perform image processing to make the image favorable for human viewing. For example, the ISP 200 can perform image processing (e.g., changing Bayer pattern image data to YUV or RGB format) to change the data format of the image data IDTA, or can perform image processing to improve image quality (e.g., noise elimination, brightness adjustment, and sharpness adjustment). In Figure 1 FIG. Figure 1 , the image signal processor ISP 200 is shown as being placed separately from the image sensor 100, but is not limited thereto, and a part of the ISP 200 can be located inside the image sensor 100.

[0034] The ISP 200 can include an image generator 210. The image generator 210 can know in advance the arrangement (e.g., spatial arrangement) and quantity of the image data set IDTA to be sensed by the image sensor 100 through a received mode signal MODE_S. For example, by receiving a mode signal MODE_S indicating a first mode, the image generator 210 can know in advance that 9 first images will be received, where the image data IDTA is arranged in a 3×3 matrix form.

[0035] The image generator 210 can determine the order of capturing the image data IDTA based on an index IDX, and can extract the spatial coordinates (i.e., the rotation degree of the prism) of the image data IDTA based on the capturing order and the known position of the optical device 110 associated with each group of image data IDTA.

[0036] The image generator 210 can include parameter data 310 stored in a memory 300. According to an exemplary embodiment, the parameter data 310 can include parameters for preprocessing or postprocessing the image data IDTA, e.g., parameters related to a homography transformation, parameters related to lens distortion correction, and parameters related to image warping. The image generator 210 can output the address ADDR of the space in which the parameter data 310 stored in the memory 300 is stored based on the mode signal MODE_S, and thus, can provide a parameter PRMT. For example, the ISP 200 can provide an address ADDR for calling a plurality of parameters PRMT to the memory 300 based on the mode signal MODE_S, and the memory 300 can provide a plurality of parameters PRMT corresponding to the mode signal MODE_S to the ISP 200 based on the address ADDR.

[0037] The image generator 210 may determine the arrangement, shooting order, and spatial coordinates (i.e., the rotation degree of the optical device 110) of the image data IDTA based on the pattern signal MODE_S and the index IDX, and may splice a plurality of image data sets IDTA into the resulting image data IDTA_H with a high viewing angle by using the parameter PRMT corresponding to the rotation degree of the optical device 110.

[0038] A large amount of computation is required to perform homography transformation, lens distortion correction, and image warping in real time. According to one technical concept of the present disclosure, by loading the parameter (PRMT) corresponding to the pattern signal (MODE_S) (e.g., the homography transformation value, lens distortion transformation value, and image warping table associated with the specific optical device position of each image data set IDTA), the computational process accompanying the estimation process (i.e., the transformation process) of the parameter PRMT can be omitted, and the computational amount of the ISP 200 can be reduced. Therefore, the image generator 210 according to an exemplary embodiment of the present disclosure can quickly merge a plurality of image data sets IDTA by loading the pre-stored parameter PRMT without performing in real time the parameter estimation for the post-processing or conversion of the image data IDTA. The merging can be automatically performed by the image generator 210 without user interaction, external software, or complex real-time parameter determination computation.

[0039] The image generator 210 may generate a high-viewing-angle image IDTA_H having a size larger than the image sensed by the image sensor 100 by splicing a plurality of image data sets IDTA. The image generator 210 according to the present invention may obtain a second image having a wide viewing angle by merging the first images successively sensed by the first image sensor having a narrow viewing angle. The second image generated by merging the images having a narrow viewing angle may have a higher resolution than the third image sensed by the second image sensor having the same resolution as the first image sensor but having a wide viewing angle.

[0040] The memory 300 may be implemented as a volatile memory or a non-volatile memory. The volatile memory may include a dynamic random access memory (DRAM), a static RAM (SRAM), etc., and the non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), and an electrically erasable programmable ROM, a flash memory, a phase change RAM (FRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. According to an exemplary embodiment, the memory 300 is included in the camera module 10 and may be a one-time programmable (OTP) memory. Parameter data 310 including a homography transformation value, a lens distortion transformation value, an image distortion table, etc. may be stored in a specific area of the internal space of the memory 300. When an address ADDR of a specific area storing the parameter PRMT is received, the memory 300 may provide the parameter PRMT stored in the corresponding storage area.

[0041] The parameter data 310 may store parameters PRMT for various situations corresponding to the mode signal MODE_S. For example, the parameter PRMT may include: a first parameter for transforming image data IDTA arranged in a 3×3 matrix form with reference to an index IDX in response to a mode signal MODE_S indicating a first mode; and a second parameter for transforming image data IDTA arranged in a 5×5 matrix form with reference to an index IDX in response to a mode signal MODE_S indicating a second mode. However, the present invention is not limited thereto, and various parameters for transforming image data IDTA arranged in various forms may be pre-calculated and stored.

[0042] According to the camera module 10 according to an embodiment of the present disclosure, a wide-angle image may be obtained by combining successively sensed narrow-angle images. The combined wide-angle image may have a higher resolution than an image captured by a device that captures a wide-angle image and has the same number of pixels as the device that captures a narrow-angle image.

[0043] In addition, compared with an imaging device having the same size as the image sensor of the imaging device according to an embodiment of the present disclosure but having a larger number of pixels than the imaging device according to an embodiment of the present disclosure, the imaging device according to an embodiment of the present disclosure may sufficiently ensure an internal pixel pitch, thereby reducing the occurrence of crosstalk and improving the SNR. In addition, the imaging device according to an embodiment of the present disclosure may skip a calculation step by loading pre-calculated and stored homography transformation, lens distortion correction, and image distortion transformation, thereby increasing the speed of image generation (or processing).

[0044] Figure 2This is a diagram of the optical device 110 according to an exemplary embodiment of the present disclosure.

[0045] Referring to Figure 2 , the optical device 110 may include an x-axis, a y-axis, and a z-axis, that is, a total of three spatial rotation axes. The optical device 110 may sense an object by rotating around at least one of the x-axis, the y-axis, and the z-axis.

[0046] Rotation around the x-axis of the optical device 110 may be referred to as roll (horizontal rocking). Rotation around the y-axis of the optical device 110 may be referred to as pitch (vertical rocking). Rotation around the z-axis of the optical device 110 may be referred to as yaw.

[0047] According to an exemplary embodiment, rotation in the pitch direction may be understood as rotation in the x-axis direction passing through the prism filter of the image sensor 100 in the horizontal direction, rotation in the yaw direction may be understood as rotation in the z-axis direction passing vertically through the prism filter of the image sensor 100, and rotation in the roll direction may be understood as rotation in the y-axis direction passing through the prism filter of the image sensor in the longitudinal direction.

[0048] The image sensor 100 may have a threshold viewing angle for capturing an object. The threshold viewing angle may be limited by a user's setting or may be limited by the physical limit of the image sensor 100. For example, when the image sensor 100 is embedded in the camera module 10, the threshold viewing angle of the image sensor 100 does not exceed 180 degrees.

[0049] According to an exemplary embodiment of the present disclosure, the optical device 110 may sense an object in front of the optical device 110 by using pitch and yaw. The optical device 110 may sense the left and right (periphery) of the object (center) by yaw and the up and down (periphery) of the object (center) by pitch. That is, the upper, lower, left, and right peripheral portions of the object may be sensed by pitch and yaw.

[0050] Roll may occur in the optical device 110. Roll may be independent of the rotation of the object in the up, down, left, and right directions. In this case, roll may be used to eliminate the user's hand shake, preprocessing, or postprocessing steps for accurate image correction.

[0051] Figure 3 This is a diagram showing a plurality of image data sets IDTA sensed by the optical device 110 according to an exemplary embodiment of the present disclosure. Referring to Figure 1 in conjunction with Figure 3 for reference.

[0052] Referring to Figure 3 , the image sensor 100 may output image data IDTA and an index IDX based on a mode signal MODE_S.

[0053] The image sensor 100 may check the arrangement (and / or quantity) of the image data set IDTA to be sensed by the image sensor 100 by receiving a mode signal MODE_S. For example, when the mode signal MODE_S indicating the first mode is received, the optical device 110 may rotate up, down, left, and right with respect to the front side of the object. Accordingly, the image sensor 100 may sense a total of nine image data sets IDTA, where three image data sets are in the horizontal direction and three image data sets are in the vertical direction.

[0054] According to an embodiment of the present disclosure, the image data IDTA may include fixed geometric characteristic information of the optical device 110. Specifically, according to the mode signal MODE_S, the arrangement of the image data IDTA obtained through the rotation range of the optical device 110 may be different. However, since the mode signal MODE_S also indicates the rotation of the optical device 110 within a predetermined range and a set of positions, the geometric characteristic information (e.g., pitch, roll, etc.) of the image data IDTA obtained through the rotation may always be constant.

[0055] According to an exemplary embodiment of the present disclosure, the image data IDTA of a 3×3 matrix array may be sequentially sensed based on the mode signal MODE_S indicating the first mode. The optical device 110 may rotate left once and then rotate right through the center, and the image sensor 100 may sense three images during the first horizontal rotation. After that, the optical device 110 may rotate down once and then rotate right from left to right through the center again, and the image sensor 100 may sense three images during the second horizontal rotation. Similarly, the optical device 110 may rotate down once more and then rotate from left to right, and the image sensor 100 may sense three images during the third horizontal rotation.

[0056] The image sensor 100 may generate nine image data sets IDTA of a sensed 3×3 array and may provide an index IDX from 1 to 9 according to the rotation order of the optical device 110. Since there are nine image data sets IDTA, the numbering from the first frame to the ninth frame may be numbered in the order from the upper left to the lower right. In this case, the fifth frame may be present in front of the image sensor 100 and may be the sensed frame, and the fourth frame may have the same pitch value as the fifth frame but may have a larger roll value. The second frame has the same roll value as the fifth frame but may have a larger pitch value, and so on. Ordinal numbers such as "first", "second", "third", etc. may be used herein as labels for certain elements, steps, etc. to distinguish these elements, steps, etc. from each other. Terms not described using "first", "second", etc. in the specification may still be referred to as "first" or "second" in the claims. Additionally, a term referred to by a specific ordinal number (e.g., "first" in a specific claim) may be described elsewhere with a different ordinal number (e.g., "second" in the specification or another claim) depending on the context.

[0057] According to an exemplary embodiment of the present disclosure, the spatial coordinates of multiple image data sets IDTA may be given in the form of (Y, P). For example, the spatial coordinates of the first image data may be (Y1, P1), the spatial coordinates of the second image data may be (Y2, P2), the spatial coordinates of the third image data may be (Y3, P3), the spatial coordinates of the fourth image data may be (Y4, P4), the spatial coordinates of the fifth image data may be (Y5, P6), the spatial coordinates of the sixth image data may be (Y6, P6), the spatial coordinates of the seventh image data may be (Y7, P7), the spatial coordinates of the eighth image data may be (Y8, P8), and the spatial coordinates of the ninth image data may be (Y9, P9).

[0058] Since the image sensor 100 senses an object while rotating, when capturing an image, an object on the side rather than in front may be distorted. To solve this problem, a homography transformation may be performed, but it requires a large amount of calculation. Therefore, according to some embodiments, the image sensor 100 may always determine the array of the sensed image based on the same fixed geometric image structure (e.g., based on the same set of preset optical device positions) and may pre-compute the homography transformation value and store the homography transformation value in a memory. The ISP 200 may omit or greatly reduce the calculation process by using the fixed geometric characteristics for the sensed image. Therefore, the amount of calculation required to perform the homography transformation may be reduced, or there may be no calculation amount at all, and the image generation or readout speed may be increased.

[0059] The ISP 200 may calculate a yaw value or a pitch value between frames, load a homography transformation value corresponding to a difference compared to the parameter data 310 stored in the memory 300, and convert an image according to the transformation value.

[0060] Figure 4A , Figure 4B and Figure 4C is a diagram illustrating a method of rotating an optical device 110 according to an example embodiment of the present disclosure. Figure 3 different, Figure 4A , Figure 4B and Figure 4C FIG. 4 shows a case where image data IDTAa, IDTAb, and IDTAc of a 5×5 matrix array are generated by receiving a mode signal MODE_S indicating a second mode. Figure 1 and Figure 4A , Figure 4B and Figure 4C Refer to it together.

[0061] refer to Figure 4A , the optical device 110a may rotate a total of five times in the first horizontal rotation to capture image data sets from left to right, and the image sensor 100 may sense first to fifth image data IDTA_1 to IDTA_5 at five corresponding positions. Thereafter, the optical device 110a may rotate downward once, and then rotate a total of five times in the second horizontal rotation to capture image data sets from left to right. The image sensor 100 may sense sixth to tenth image data IDTA_6 to IDTA_10 at five corresponding positions.

[0062] Similarly, the optical device 110a can rotate five times in the third horizontal rotation, the fourth horizontal rotation, and the fifth horizontal rotation, respectively, to capture additional data sets from left to right. Therefore, the image sensor 100 can sequentially sense the eleventh image data IDTA_11 to the fifteenth image data IDTA_15, the sixteenth image data IDTA_16 to the twentieth image data IDTA_20, and the twenty-first image data IDTA_21 to the twenty-fifth image data IDTA_25.

[0063] because Figure 4A The method of sequentially sensing from left to right and from top to bottom shown in is similar to a raster scanning method and is a conventional scanning method in the field of image processing, so it may be relatively easy to apply an algorithm to process image data.

[0064] refer to Figure 4B, the optical device 110b can rotate a total of five times in the first horizontal rotation to capture an image dataset, and the image sensor 100 can sense the first image data IDTA_1 to the fifth image data IDTA_5. After that, the optical device 110b can rotate downward once, and then rotate four times from right to left in the second horizontal rotation, and the image sensor 100 can sense the sixth image data IDTA_6 to the tenth image data IDTA_10.

[0065] Similarly, the optical device 110b can rotate four times from left to right in the third horizontal rotation, four times from right to left in the fourth horizontal rotation, and four times from left to right in the fifth horizontal rotation. Accordingly, the image sensor 100 can sequentially generate the eleventh image data IDTA_11 to the fifteenth image data IDTA_15, the sixteenth image data IDTA_16 to the twentieth image data IDTA_20, and the twenty-first image data IDTA_21 to the twenty-fifth image data IDTA_25.

[0066] Compared with Figure 4A the Figure 4B rotation method of the optical device 110b shown in Figure 4A may be path-efficient. More specifically, according to the rotation method shown in Figure 4B , after rotating from left to right, when rotating in the horizontal direction for the next row, image data can be obtained only after the optical device 110a that has rotated to the right rotates to the left. In contrast,

[0067] the rotation method shown in

[0068] uses the already rotated optical device 110b as it is, but rotates only one row in the vertical direction, so that the horizontal rotation path (or time) of the optical device 110b can be saved.

[0067] The image sensor 100 senses a relatively large amount of image data IDTA (e.g., 25) in a short time. As the sensing time increases, the image data IDTA may be distorted due to external factors such as the movement of an object and the jitter of the image sensor. According to the technical concept of the present disclosure, the distortion of the image data IDTA can be minimized by sensing an image with an efficient path.

[0068] Refer to Figure 4C, the image sensor 100 may sense image data IDTA starting from the center of a scene where an object of interest is more likely to be present. According to an exemplary embodiment, when the optical device 110c does not rotate, the image sensor 100 may sense first image data IDTA_1. Thereafter, the second image data IDTA_2 may be sensed by rotating the optical device 110c left once, and then the third image data IDTA_3 may be sensed by rotating the optical device 110c up once. The fourth image data IDTA_4 and the fifth image data IDTA_5 may be sensed by rotating the optical device 110c right twice again, and then the sixth image data IDTA_6 and the seventh image data IDTA_7 may be sensed by rotating the optical device 110c down twice again.

[0069] Similarly, by rotating the optical device 110c left three times, the eighth image data IDTA_8, the ninth image data IDTA_9, and the tenth image data IDTA_10 may be generated. And then, by rotating the optical device 110c up three times, the eleventh image data IDTA_11, the twelfth image data IDTA_12, and the thirteenth image data IDTA_13 may be generated.

[0070] According to the same principle, the fourteenth image data IDTA_14, the fifteenth image data IDTA_15, the sixteenth image data IDTA_16, and the seventeenth image data IDTA_17 may be generated by rotating the optical device 110c right four times, and then the eighteenth image data IDTA_18, the nineteenth image data IDTA_19, the twentieth image data IDTA_20, and the twenty - first image data IDTA_21 may be generated by rotating the optical device 110c down four times. And then, the twenty - second image data IDTA_22, the twenty - third image data IDTA_23, the twenty - fourth image data IDTA_24, and the twenty - fifth image data IDTA_25 may be generated by rotating the optical device 110c left four times.

[0071] The above process may be described as the optical device 110c rotating clockwise or in a spiral manner. Although not shown in Figure 4C , the optical device 110c may also rotate in the counter - clockwise direction.

[0072] Compared with Figure 4A or Figure 4B , Figure 4CThe method of rotating the optical device 110c shown can be directed towards an object of interest. Generally, the object is highly likely to be located at the center of the scene. As the sensing time increases, the image data (IDTA) may be distorted due to external factors such as object movement and image sensor jitter. Therefore, by starting sensing from the center of the scene within a relatively short time, the distortion of the information about the object of interest can be minimized.

[0073] Figure 5 FIG. is a diagram illustrating a method of generating an image in an image generator 210 according to an exemplary embodiment of the present disclosure. Refer Figure 1 to Figure 5 together.

[0074] Referring Figure 5 to, the image generator 210 can generate high-resolution image data IDTA_H with a relatively wide viewing angle by combining multiple image data sets IDTA.

[0075] The image generator 210 can combine (i.e., stitch) multiple image data sets IDTA based on a mode signal MODE_S and an index IDX.

[0076] According to an exemplary embodiment, since the first image data IDTA_1 and the second image data IDTA_2 are sensed at a relatively narrow viewing angle for the same scene, the included images may be different, but there may be some overlapping regions. The image generator 210 can use the overlapping regions between the first image data IDTA_1 and the second image data IDTA_2 by correcting the distortion and performing geometric transformation on each of the first image data IDTA_1 and the second image data IDTA_2 to combine the two image data.

[0077] Similarly, the image generator 210 can combine the second image data IDTA_2 and the third image data IDTA_3. Therefore, the image generator 210 can generate high-resolution image data IDTA_H with a wide viewing angle. The high-resolution image data IDTA_H generated by combining images with a relatively narrow viewing angle can have a higher resolution than the image data IDTA sensed by an image sensor having the same resolution but a wide viewing angle.

[0078] Geometric transformation and distortion correction are generally required to combine multiple image data IDTA. A large amount of computation is usually used to transform and correct multiple image data IDTA in real time, which reduces the image processing speed. The camera module or the image processing method according to certain embodiments of the present disclosure can minimize the amount of computation by referring to parameters PRMT previously stored based on fixed geometric characteristics.

[0079] Figure 6It is a block diagram showing processing operations in an image signal processor according to an exemplary embodiment of the present disclosure.

[0080] The ISP 200 may preprocess or postprocess the sensed image data IDTA. The ISP 200 may include a preprocessing module, a lens distortion correction module, a homography calculation module, an image warping module, a motion compensation module, a preprocessing or postprocessing module, and a cropping module.

[0081] Preprocessing may refer to appropriately performing preprocessing for applying and using an image enhancement algorithm to image artifacts. For example, preprocessing may be a preprocessing operation for eliminating distortion of the sensed image data IDTA and improving algorithm performance, such as, white balance, denoising, demosaicing, lens shading, gamma correction, edge detection, edge enhancement, etc.

[0082] Postprocessing may refer to applying an image enhancement algorithm to image artifacts. For example, postprocessing may be a postprocessing operation for eliminating distortion of the sensed image data IDTA, such as, white balance, denoising, demosaicing, lens shading, gamma correction, etc.

[0083] Lens distortion correction may refer to correction of a situation where, when an abnormality occurs on the lens surface or refraction occurs, an image caused by the lens projection center cannot be accurately formed at the focal point. For example, image distortion may occur in the edge region of the optical device 110.

[0084] In an exemplary embodiment, lens distortion correction may be performed by Equation 1 below.

[0085] [Equation 1]

[0086]

[0087] Here, k1, k2, and are parameters related to the radial aberration of the distortion aberration, and p1, p2, u u and u v are parameters related to decentered distortion.

[0088] In order to separately correct multiple image data sets IDTA, calculations need to be performed among many parameters shown in Equation 1. According to the technical concept of the present disclosure, since the geometric characteristics of the image data set IDTA are fixed, the ISP 200 may omit complex calculation steps by referring to the first parameter PRMT1 related to lens distortion correction in the parameter data 310 pre-calculated and stored in the memory 300, as if actually performing correction. For ease of explanation, a method for correcting lens distortion has been shown, but it should be understood that various lens distortion correction methods may be applied.

[0089] Homography is a term that can refer to a mapping relationship or transformation relationship established between the 3D camera coordinates or 2D image coordinates of a planar object. More specifically, homography can refer to a linear transformation relationship between two points captured at two different positions of a planar object in three-dimensional space. After searching for corresponding points in two images, parameters necessary for homography operations can be extracted based on the corresponding points.

[0090] In an exemplary embodiment, homography can be represented by Equation 2 below.

[0091] [Equation 2]

[0092]

[0093] Here, h 00 , h 01 , h 02 , h 10 , h 11 , h 12 , h 20 , h 21 , h 22 are homography parameters. According to the technical concept of the present disclosure, since the geometric characteristics of the image data set IDTA are fixed, the ISP 200 can omit complex operation steps for calculating homography by referring to the second parameter PRMT2 related to homography in the parameter data 310 stored in the memory 300 as an actual correction. The ISP 200 can load the homography parameters (i.e., homography coefficients) stored in the memory 300 to apply the homography transformation value to the sensed image.

[0094] Image warping is projecting images at different angles onto an image plane. Image warping can be represented by Equation 3 below.

[0095] [Equation 3]

[0096]

[0097] Here, a, b, c, d, e, f, g, and h are warping parameters, x and y are the original coordinates, and wx and wy are the warped coordinates.

[0098] According to the technical concept of the present disclosure, since the geometric characteristics of the image data set IDTA are fixed, the ISP 200 can omit complex operation steps for calculating image warping by referring to the third parameter PRMT3 related to image warping in the parameter data 310 stored in the memory 300 as an actual correction.

[0099] Here, the parameter data 310 may be stored in a common area of the memory 300, or may be separated and stored in different areas. Additionally, it is not excluded that the memory 300 may include multiple memory devices.

[0100] Thereafter, motion compensation for the shake of the camera module is performed, and then post-processing of the image processor is performed. The post-processing may refer to applying an image enhancement algorithm to image artifacts. For example, the post-processing may include various processing methods for eliminating the distortion of the sensed image data IDTA and performing additional correction, such as white balance and denoising, demosaicing, lens shading, gamma correction, etc.

[0101] Finally, by cropping each image and stitching the cropped images into one image, an image larger in size (or an image with a wide viewing angle) of the low-resolution image can be generated. For example, an image with a shallower depth (short focal length) can be generated based on an image with a deeper depth (long focal length).

[0102] Figure 7 is a flowchart showing an image processing method according to an exemplary embodiment of the present disclosure. Refer to Figure 1 and Figure 7 together.

[0103] In step S110, as a result of rotating the optical device 110 using at least one of the x-axis, y-axis, and z-axis as an axis, the image sensor 100 may generate a plurality of first images (i.e., image data IDTA) based on the mode signal MODE_S.

[0104] In step S120, the ISP 200 may correct the plurality of first images based on a first parameter corresponding to the mode signal among the plurality of parameters PRMT previously stored in the memory 300.

[0105] In step S130, a second image larger than each first image may be generated by merging the corrected first images. Each first image may correspond to the image data IDTA, and the second image may correspond to the high-resolution image data IDTA_H. The first image may have a deeper depth of field or a longer focal length than the second image.

[0106] Figure 8 is a detailed flowchart showing a correction method in an image processing method according to an exemplary embodiment of the present disclosure. Refer to Figure 1 and Figure 7 together with Figure 8 for reference.

[0107] The method S120 for correcting the image data IDTA may be subdivided as follows.

[0108] In step S121, the number and arrangement of a plurality of first images corresponding to the pattern signal MODE_S can be determined.

[0109] In step S122, the coordinates of each of the plurality of first images can be referred to. The coordinates can be related to the fixed geometric features included in the image data IDTA.

[0110] In step S123, image distortion can be eliminated by using a first parameter corresponding to the number, arrangement, and coordinates of the plurality of first images among the plurality of parameters PRMT previously stored in the parameter data 310 area of the memory 300.

[0111] Figure 9A and Figure 9B are block diagrams showing imaging devices 20a and 20b according to example embodiments of the present disclosure. Figure 1 with Figure 9A and Figure 9B be referred to together.

[0112] Compared with Figure 1 in Figure 9A and Figure 9B an application processor (AP) is further shown. The imaging devices 20a and 20b may further include AP 400a and 400b. AP 400a and 400b may be a central processing unit (CPU), a microprocessor, or a microcontroller unit (MCU), but are not limited thereto.

[0113] The ISPs 200a and 200b and the APs 400a and 400b may be implemented as a processing circuit, such as hardware including logic circuits, or may be implemented as a combination of hardware and software, such as a processor that executes software that performs image processing. In particular, the processing circuit may be implemented as a central processing unit (CPU), an arithmetic logic unit (ALU) that performs arithmetic and logical operations, shift operations, etc., a digital signal processor (DSP), a microprocessor, an application specific integrated circuit (ASIC), etc., but is not limited thereto.

[0114] Referring to Figure 9A , the imaging device 20a may include: an image sensor 100a including an optical device 110a; an ISP 200a including an image generator 210a; a memory 300a including parameter data 310a; and an AP 400a. In Figure 1 the image sensor 100, the ISP 200, the memory 300, etc. have been described above, and redundant descriptions are omitted.

[0115] In Figure 9A different from Figure 1 the camera module ( Figure 110) does not include a memory, while the imaging device 20a includes a memory. For example, Figure 9A the ISP 200a in it can combine images by referring to the memory installed inside the imaging device 20a (e.g., S-RAM and D-RAM) without referring to the memory installed on the camera module 10 (e.g., OTP memory). For example, Figure 1 the camera module 10 of can include an image sensor 100 and an optical device 110, which are physically housed in a specific housing (e.g., on a mobile phone, or in a specific compartment of a mobile phone, laptop, tablet, camera, or other device), while in Figure 9A the embodiment shown, the image sensor 100a including the optical device 110a can be placed separately from the other components shown (e.g., the image sensor 100a can be located in a separate compartment of a mobile phone, laptop, or tablet, or can be part of a camera or other device that is wirelessly or via a wire connected to a computer including the ISP 200a, AP 400a, and memory 300a).

[0116] Referring to Figure 9B , the ISP 200b can be implemented inside the AP 400b. In this case, the AP400b of the imaging device 20b can be a processor that executes software that specifically performs image processing, and the ISP 200b can be implemented as software or a combination of hardware and software.

[0117] Figure 10A and Figure 10B are block diagrams showing an imaging device including a plurality of camera modules according to an exemplary embodiment of the present disclosure.

[0118] Referring to Figure 10A , the electronic device 20000 can include a multi-camera module 1100, an AP 4000, and a memory 5000. The memory 5000 can perform the same functions as Figure 1 the memory 300 shown, and redundant descriptions will be omitted. Figure 10A At least one of the camera modules 1100a, 1100b, and 1100c of can perform functions similar to those of Figure 1 the camera module 10 of.

[0119] The electronic device 20000 can capture and / or store an image of an object using a CMOS image sensor, and can be implemented as a mobile phone, tablet, or portable electronic device. The portable electronic device can include a laptop, mobile phone, smart phone, deck PC, wearable device, etc. The electronic device 20000 can include one or more camera modules and an AP that processes image data generated from the one or more camera modules.

[0120] The multi-camera module 1100 may include a first camera module 1100a, a second camera module 1100b, and a third camera module 1100c. The multi-camera module 1100 may perform the same functions as Figure 1 the camera module 10. Although three camera modules 1100a to 1100c are shown for ease of description, the present invention is not limited thereto, and various numbers of camera modules may be included in the multi-camera module 1100.

[0121] The viewing angle or field of view of the first camera module 1100a may be greater than that of the second camera module 1100b. According to an exemplary embodiment of the present disclosure, the second camera module 1100b may include an optical device ( Figure 1 110), and may generate a plurality of image data sets IDTA by rotating the optical device 110 and stitch the image data sets IDTA by using pre-stored parameters PRMT. Accordingly, the second camera module 1100b may generate a high-resolution image having a viewing angle similar to that of the first camera module 1100a but having a relatively high resolution.

[0122] Hereinafter, the detailed configuration of the second camera module 1100b will be described in more detail with reference to Figure 10B However, according to an embodiment, the following description may be equally applicable to the other camera modules 1100a and 1100c.

[0123] Referring to Figure 10B , the second camera module 1100b may include a prism 1105, an optical path folding element (hereinafter simply referred to as "OPFE") 1110, an actuator 1130, an image sensing device 1140, and a storage unit 1150.

[0124] The prism 1105 may change the path of light L incident from the outside of a reflection surface 1107 including a light reflecting material.

[0125] According to an exemplary embodiment, the prism 1105 may change the path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. Additionally, the prism 1105 may rotate the reflection surface 1107 of the light reflecting material relative to the central axis 1106 in the A direction, or may rotate the central axis 1106 in the B direction, thereby changing the path of light L incident in the first direction X to the perpendicular second direction Y. In this case, the OPFE 1110 may move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0126] In an exemplary embodiment, as shown in the figure, the maximum rotation angle of the prism 1105 in the A direction is equal to or less than 21 degrees in the positive (+) A direction, and can be greater than 21 degrees in the negative (-) A direction, but the embodiment is not limited thereto.

[0127] In an exemplary embodiment, the prism 1105 can be moved by about 20 degrees in the positive (+) or negative (-) B direction, or between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees. Here, the movement angle can be moved to the same angle in the positive (+) or negative (-) B direction, or can be moved to approximately similar angles within a range of about 1 degree, but is not limited thereto.

[0128] In an exemplary embodiment, the prism 1105 can move the reflection surface 1107 of the light reflecting material in a third direction (e.g., the Z direction) parallel to the extension direction of the central axis 1106.

[0129] The OPFE 1110 can include, for example, an optical lens composed of m groups (where m is a natural number). The m lenses can be moved in the first direction X to change the optical zoom ratio of the second camera module 1100b. For example, assuming that the default optical zoom magnification of the second camera module 1100b is Z, when the m optical lenses included in the OPFE 1110 are moved, the optical zoom magnification of the second camera module 1100b can be changed to 3Z, 5Z, or greater than 5Z.

[0130] The actuator 1130 can move the OPFE 1110 or the optical lens (hereinafter simply referred to as the optical lens) to a specific position. For example, the actuator 1130 can adjust the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for precise sensing.

[0131] The image sensing device 1140 can include an image sensor 1142, control logic 1144, and a memory 1146. The image sensor 1142 can sense an image of a sensing target using the light L provided by the optical lens. Since Figure 10B the image sensor 1142 can be functionally similar to Figure 1 the image sensor 100, redundant descriptions are omitted. The control logic 1144 can control the overall operation of the second camera module 1100b. For example, the control logic 1144 can control the operation of the second camera module 1100b according to a control signal provided through the control signal line CSLb.

[0132] The memory 1146 may store information for the operation of the second camera module 1100b, for example, calibration data 1147. The calibration data 1147 may include information necessary for the second camera module 1100b to generate image data using the light L provided from the outside. The calibration data 1147 may include, for example, information about the degree of rotation, information about the focal length, information about the optical axis, and calibration information for image processing. When the second camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 1147 may include the focal length value for each position (or state) of the optical lens and information related to autofocus.

[0133] The storage unit 1150 may store the image data sensed by the image sensor 1142. The storage unit 1150 may be disposed outside the image sensing device 1140 and may be implemented in the form of being stacked with the sensor chip constituting the image sensing device 1140. In an exemplary embodiment, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiment is not limited thereto.

[0134] In an exemplary embodiment, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

[0135] In an exemplary embodiment, one of the plurality of camera modules 1100a, 1100b, and 1100c (for example, the second camera module 1100b) is a folded lens type camera module including the above-described prism 1105 and OPFE 1110, and the remaining camera modules (for example, 1100a and 1100c) may be vertical type camera modules that do not include the prism 1105 and OPFE 1110, but the embodiment is not limited thereto.

[0136] In an exemplary embodiment, one of the plurality of camera modules 1100a, 1100b, and 1100c (for example, the third camera module 1100c) may be a vertical type depth camera that extracts depth information by using infrared rays (IR), for example. In this case, the AP 4000 may generate a 3D depth image by combining the image data provided from such a depth camera with the image data provided from another camera module (for example, the first camera module 1100a or the second camera module 1100b).

[0137] In an exemplary embodiment, at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., the first camera module 1100a or the second camera module 1100b) may have different fields of view (angles of view) from each other. In this case, for example, the optical lenses of at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., the first camera module 1100a or the second camera module 1100b) may be different from each other, but the present invention is not limited thereto. For example, among the plurality of camera modules 1100a, 1100b, and 1100c, the first camera module 1100a may have a smaller field of view (FOV) than the second camera module 1100b and the third camera module 1100c. However, the present invention is not limited thereto, and the multi-camera module 1100 may further include a camera module having a larger field of view (FOV) than the originally used camera modules 1100a, 1100b, and 1100c.

[0138] In addition, in some embodiments, the viewing angles of each of the plurality of camera modules 1100a, 1100b, and 1100c may be different. In this case, the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, but the present invention is not limited thereto.

[0139] In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be physically separated from each other. For example, in one embodiment, the sensing area of one image sensor 1142 is not divided and used by the plurality of camera modules 1100a, 1100b, 1100c, but independent image sensors 1142 may be placed inside each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0140] The AP 4000 may include: a plurality of sub-processors 4100a, 4100b, and 4100c; a decoder 4200; a camera module controller 4300; a memory controller 4400; and an internal memory 4500.

[0141] The AP 4000 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the AP 4000 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips by being separated from each other.

[0142] Image data generated by each of the camera modules 1100a, 1100b, and 1100c can be provided to the corresponding sub-processors 4100a, 4100b, and 4100c via the mutually separated image signal lines ISLa, ISLb, and ISLc. For example, the image data generated by the first camera module 1100a can be provided to the first sub-processor 4100a via the first image signal line ISLa, the image data generated by the second camera module 1100b can be provided to the second sub-processor 4100b via the second image signal line ISLb, and the image data generated by the third camera module 1100c can be provided to the third sub-processor 4100c via the third image signal line ISLc. Such image data transmission can be performed using, for example, a camera serial interface (CSI) based on the Mobile Industry Processor Interface (MIPI), but the embodiments are not limited thereto.

[0143] In an exemplary embodiment, one sub-processor can be arranged to correspond to multiple camera modules. For example, the first sub-processor 4100a and the third sub-processor 4100c are not implemented separately from each other as shown in the figure, but are implemented as a single sub-processor, and the image data provided from the first camera module 1100a and the third camera module 1100c can be selected (e.g., by a multiplexer or the like) and then provided to the integrated sub-image processor.

[0144] The camera module controller 4300 can provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals provided by the camera module controller 4300 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c via the mutually separated control signal lines CSLa, CSLb, and CSLc.

[0145] Any one of the multiple camera modules 1100a, 1100b, and 1100c can be designated as a main camera (e.g., 1100b) according to image generation information including a zoom signal or a mode signal, and the remaining camera modules (e.g., 1100a and 1100c) can be designated as slave cameras. Such information can be included in the control signal and provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0146] Under the control of the camera module controller 4300, the camera modules 1100a, 1100b, and 1100c that operate as the main camera and the sub - camera can be changed. For example, when the viewing angle of the first camera module 1100a is greater than the viewing angle of the second camera module 1100b and the zoom factor indicates a low zoom ratio, the second camera module 1100b can operate as the main camera, and the first camera module 1100a can operate as the sub - camera. Conversely, when the zoom factor indicates a high zoom ratio, the first camera module 1100a can operate as the main camera, and the second camera module 1100b can operate as the sub - camera.

[0147] In an exemplary embodiment, the control signals provided by the sub - camera module controller 4300 to each of the camera modules 1100a, 1100b, and 1100c can include a synchronization enable signal. For example, when the second camera module 1100b is the main camera and the first camera module 1100a and the third camera module 1100c are sub - cameras, the camera module controller 4300 can send a synchronization enable signal to the second camera module 1100b. The second camera module 1100b that receives such a synchronization enable signal can generate a synchronization signal based on the provided synchronization enable signal and can provide the generated synchronization signal to the first camera module 1100a and the third camera module 1100c through the synchronization signal line SSL. The first camera module 1100b, the second camera module 1100a, and the third camera module 1100c can send image data to the AP 4000 in synchronization with the synchronization signal.

[0148] Although the inventive concept has been specifically shown and described with reference to embodiments of the inventive concept, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A camera module, comprising: An image sensor including an optical device configured to capture an image by rotating about at least one of x-axis, y-axis, and z-axis perpendicular to each other within a specific angle at a specific position in response to a mode signal, and the image sensor is configured to generate a plurality of first images for generating a single second image, each of the plurality of first images being generated when the optical device is at a different position; And An image signal processor ISP configured to process the plurality of first images to generate the second image, Wherein the ISP is further configured to obtain a parameter corresponding to the mode signal from a plurality of pre-stored parameters according to the mode signal to correct the plurality of first images, and generate the second image by combining the corrected first images.

2. The camera module according to claim 1, wherein, The plurality of parameters includes at least one of a homography parameter, a lens distortion correction parameter, and a skew parameter.

3. The camera module according to claim 2, wherein, The plurality of parameters includes the homography parameter, and the homography parameter is pre-determined with respect to the number of the plurality of first images, the arrangement of the plurality of first images, and the coordinates of each of the plurality of first images determined based on the mode signal.

4. The camera module according to claim 1, wherein, In terms of the number of pixels, the size of the second image is larger than the size of each of the plurality of first images.

5. The camera module according to claim 1, wherein, The ISP is further configured to obtain arrangement information of the plurality of first images combined to form the second image by receiving the mode signal.

6. The camera module according to claim 1, wherein, The image sensor is configured to sense an image using at least one of pitching, rolling, and yawing.

7. The camera module according to claim 1, wherein, The optical device includes a prism or a mirror.

8. The camera module according to claim 7, wherein, The optical device is further configured to generate the plurality of first images by pitching and rotating horizontally from left to right and by rolling and rotating vertically from top to bottom.

9. The camera module according to claim 7, wherein, The optical device is further configured to sense the plurality of first images by rotating outward from the center of a scene where an object to be sensed exists, either clockwise or counterclockwise.

10. The camera module according to claim 1, further comprising a memory, Among them, The plurality of parameters are stored in the memory.

11. The camera module according to claim 10, wherein, The ISP is further configured to provide an address for loading the plurality of parameters to the memory based on the mode signal, and The memory is configured to provide the parameter corresponding to the mode signal to the ISP based on the address.

12. An image processing method of a camera module capable of generating an image with a large viewing angle from an image with a small viewing angle, the image processing method comprising: In response to a mode signal, generating a plurality of first images by rotating an optical device about at least one of x-axis, y-axis, and z-axis perpendicular to each other within a specific angle at a specific position, each of the plurality of first images being generated when the optical device is at a different position, and the optical device uses a lens or a mirror; Correcting the plurality of first images based on a first parameter corresponding to the mode signal among a plurality of pre-stored parameters; And Generating a second image with a size larger than each of the plurality of first images by combining the corrected first images.

13. The image processing method according to claim 12, wherein, Correcting the plurality of first images includes: Transform the plurality of first images into perspective projections based on the homography parameters; Correct the distortion of the optical device based on the lens distortion correction parameters; and Change the form of the plurality of first images based on the warping parameters.

14. The image processing method according to claim 13, wherein, The homography parameters are predetermined with respect to the number and arrangement of the plurality of first images, and with respect to the coordinates of each of the plurality of first images determined based on the pattern signal.

15. The image processing method according to claim 12, wherein, Correcting the plurality of first images includes: Determining the number and arrangement of the plurality of first images corresponding to the pattern signal; Referring to the coordinates of each of the plurality of first images; and Eliminating image distortion by using the first parameter corresponding to the number, arrangement, and coordinates among the plurality of pre-stored parameters.

16. An imaging device, comprising: A first image sensor configured to generate a plurality of first images by rotating an optical device using a lens or a mirror at least in one direction of pitch, roll, and yaw within a specific angle at a specific position based on a first pattern signal; A second image sensor configured to generate a large-view image having a larger viewing angle than the viewing angles of the plurality of first images; And An image signal processor ISP configured to process the plurality of first images, wherein the ISP is further configured to correct the plurality of first images based on the parameters corresponding to the first pattern signal among the plurality of pre-stored first parameters, and generate a second image by combining the corrected first images.

17. The imaging device according to claim 16, wherein, The second image has a viewing angle larger than the viewing angle of the large-view image or the same viewing angle as the large-view image, but has a higher resolution than the large-view image.

18. The imaging device according to claim 17, wherein, The first parameters include homography parameters, lens distortion correction parameters, and warping parameters.

19. The imaging device according to claim 17, wherein, The first image sensor is configured to generate 9 first images arranged in a 3×3 matrix array in response to the first pattern signal, and The ISP is configured to generate the second image by referring to the parameters corresponding to the first pattern signal among the plurality of first parameters.

20. The imaging device according to claim 16, wherein the first image sensor is configured to generate 25 first images arranged in a 5×5 matrix array in response to a second pattern signal, and The ISP is configured to generate a third image having a larger viewing angle than the viewing angle of the second image by referring to second parameters corresponding to the second pattern signal.

Images