CAMERA DEVICE AND ELECTRONIC DEVICE WITH IT

Description

BACKGROUND OF THE DISCLOSURE 1. Field of the disclosure

[0001] The present disclosure relates to a camera device and an electronic device including the same, and more particularly to a camera device capable of rapidly and accurately compensating for a difference between a color image from a color camera and an IR image from an IR camera, and an electronic device including the same.2. Description of the Related Art

[0002] A camera device is an image capturing device. Recently, cameras are employed in various electronic devices.

[0003] Meanwhile, the camera device may include a plurality of cameras.

[0004] For example, the camera device may include two color cameras or may include one color camera and one IR camera.

[0005] Meanwhile, in the case where the camera device includes the color camera and the IR camera, the camera device may perform signal processing by combining a color image, obtained from the color camera, with an IR image from the IR camera. In this case, calibration is required between the color image and the IR image.

[0006] Meanwhile, in the case where the camera device is mounted in an electronic device, if there is no bracket for fixing the color camera and the IR camera in order to allow freedom of design of the electronic device, a distance and the like between the color camera and the IR camera may change irregularly, in which case calibration is further required between the color image and the IR image.

[0007] US 2009 / 201384 A1 discloses a method and apparatus for matching images. In the document the image matching method is described to include obtaining a depth image and an intensity image of an object using a depth camera installed at a first position and a color image of the object using a color camera installed at a second position other than the first poison, transforming the obtained depth image and intensity image into a depth image and an intensity image, respectively, that could be obtained if the object were photographed by a camera at the second position, and matching the transformed depth image and intensity image and the obtained color image.

[0008] US 2018 / 315213 A1 discloses an apparatus for calibrating texture cameras. In the document the apparatus is described to include an image receiver to receive a depth image from a depth camera and a color image from a texture camera, a feature extractor to extract features from the depth image and the color image, a feature tester to detect that the extracted features from the depth image and the color image exceed a quality threshold, a misalignment detector to detect a misalignment between the extracted features from depth image and the extracted features from color image exceeds a misalignment threshold, a calibrator to modify calibration parameters for the texture camera to reduce the detected misalignment between the extracted features from the depth image and the extracted features from the color image below a misalignment threshold.

[0009] US 2013 / 010079 A1 discloses a system including a receiver component that receives a first digital image from a color camera, wherein the first digital image comprises a planar object, and a second digital image from a depth sensor, wherein the second digital image comprises the planar object, and a calibrator component that jointly calibrates the color camera and the depth sensor based at least in part upon the first digital image and the second digital image.

[0010] US 2017 / 104979 A1 discloses a system for determining a loss of calibration in a multi-view stereo imaging system including executing instructions, via a processor, for a multi-view stereo imaging system to process a plural image frame recorded from a plurality of digital cameras of an information handling system and based on plural image calibration parameters and detecting a physical impact event, via a physical sensor, to an information handling system.

[0011] US 2016 / 073102 A1 discloses a variety of techniques which include detecting features on a set of images, describing features on the set of images, determining a match between features of the image sets, determining a shift on the matched features based on camera positions associated with the matched features, determining a first homography between the camera positions and the determined shift, and determining a second homography based on a re-projection of three-dimensional features back to the cameras.SUMMARY

[0012] It is an object of the present disclosure to provide a camera device according to claim 1 capable of rapidly and accurately compensating for a difference between a color image from a color camera and an IR image from an IR camera, and an electronic device including the same.

[0013] Optional embodiments of the camera device are provided in claims 2-4.EFFECTS OF THE DISCLOSURE

[0014] A camera device and an electronic device including the same include: a color camera; an IR camera; and a processor configured to extract a first region of a color image from the color camera, to extract a second region of an IR image from the IR camera, to calculate error information based on a difference between a gradient of the first region and a gradient of the second region, to compensate for at least one of the color image and the IR image based on the calculated error information, and to output a compensated color image or a compensated IR image, thereby rapidly and accurately compensating for a difference between the color image from the color camera and the IR image from the IR camera.

[0015] Meanwhile, the processor extracts a first region of a color image from the color camera and a second region of an IR image from the IR camera; calculates three-dimensional (3D) error information based on a difference between a gradient of the first region and a gradient of the second region; based on the calculated 3D error information, compensates for at least one of the color image and the IR image to output the compensated color image or the compensated IR image. Particularly, based on the calculated 3D error information, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0016] Meanwhile, the processor normalizes the gradient of the first region and the gradient of the second region, and calculates 3D error information based on a difference between the normalized gradient of the first region and the normalized gradient of the second region; and based on the calculated 3D error information, compensates for at least one of the color image and the IR image to output the compensated color image or the compensated IR image. Particularly, by normalization, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0017] Meanwhile, the processor updates the 3D error information based on pre-stored 3D error information and the calculated 3D error information, and compensates for at least one of the color image and the IR image based on the updated 3D error information to output the compensated color image or the compensated IR image. Accordingly, by updating the 3D error information which is changed by an external impact and the like, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0018] Meanwhile, the processor upscales the IR image; may extract a first region of the color image from the color camera and a second region from the upscaled IR image; may calculate 3D error information based on a difference between a gradient of the first region and a gradient of the second region; and based on the calculated 3D error information, may compensates for at least one of the color image and the IR image to output the compensated color image or the compensated IR image. Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0019] Meanwhile, in response to movement or impact sensed by a sensor device being greater than or equal to a first level, the processor extracts the first region of the color image from the color camera and the second region of the IR image from the IR camera: and calculates error information based on the difference between the gradient of the first region and the gradient of the second region, and compensates for at least one of the color image and the IR image based on the calculated error information to output the compensated color image or the compensated IR image. Accordingly, if there is an external impact and the like, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0020] Meanwhile, the first region and the second region may include a face region. Accordingly, based on the common face region, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0021] Meanwhile, the processor may extract a first region of the color image from the color camera and a second region of the IR image from the IR camera, may calculate error information based on a difference between a gradient of a luminance component of the first region and a gradient of a luminance component of the second region, and may compensate for at least one of the color image and the IR image based on the calculated error information to output a compensated color image or a compensated IR image. Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0022] Meanwhile, the processor includes: a feature point analyzer configured to analyze each feature point of the color image from the color camera and the IR image from the IR camera; an external variable calculator configured to calculate an external variable based on each feature point information analyzed by the feature point analyzer; an error calculator configured to calculate 3D error information based on the calculated external variable; and an error corrector configured to correct an error based on the 3D error information. Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0023] Meanwhile, based on each feature point information analyzed by the feature point analyzer, the external variable calculator may calculate rotation information, translation information, or shift information of each of the color image and the IR image. Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0024] Meanwhile, based on the rotation information, the translation information, or the shift information of each of the color image and the IR image, the error calculator may calculate relative rotation information, relative translation information, or relative shift information between the color image and the IR image. Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0025] Meanwhile, the color camera and the IR camera are connected by a connection member, wherein the connection member is flexible, thereby improving a degree of freedom in designing a space of the camera device or the electronic device in which the camera device is mounted.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG. 1A is a diagram illustrating the appearance of a camera device related to the present disclosure; FIG. 1B is a diagram illustrating the appearance of a camera device according to an embodiment of the present disclosure; FIG. 2 is a diagram explaining various examples of an electronic device; FIG. 3A is an internal cross-sectional view of a color camera and an IR camera of FIG. 1B; FIG. 3B is an internal block diagram of a camera device including the color camera and the IR camera of FIG. 1B; FIGS. 4A to 5Care diagrams referred to in the description of an operation of the camera device of FIG. 3B; FIG. 6 is a flowchart illustrating an operating method of a camera device; FIGS. 7 to 10F are diagrams referred to in the description of the operating method of FIG. 6; FIGS. 11A to 11C are diagrams referred to in the description of a distortion between a color camera and an IR camera in the case where there is an impact on a camera device; FIG. 12 is a flowchart illustrating an operating method of a camera device according to an embodiment of the present disclosure; and FIGS. 13A to 13Fare diagrams referred to in the description of the operating method of FIG. 12. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings. With respect to constituent elements used in the following description, suffixes "module" and "unit" are given only in consideration of ease in the preparation of the specification, and do not carry any important meaning or role. Therefore, the suffixes "module" and "unit" may be used interchangeably.

[0028] FIG. 1A is a diagram illustrating the appearance of a camera device related to the present disclosure.

[0029] Referring to the drawing, an example is illustrated in which a camera device 100m includes a color camera CCmm, an IR camera DCmm, a bracket BRK for fixing the color camera CCmm and the IR camera DCmm, an interface CTam of the color camera CCmm, an interface CTbm of the IR camera DCmm.

[0030] Due to the bracket BRK for fixing the color camera CCmm and the IR camera DCmm in the camera device 100m according to FIG. 1A, there may be limitations in designing a space of the camera device 100m or an electronic device in which the camera device 100m is mounted.

[0031] FIG. 1B is a diagram illustrating the appearance of a camera device according to an embodiment of the present disclosure.

[0032] Referring to the drawing, an example is illustrated in which a camera device 100 includes a color camera CCm, an IR camera DCm, a connection member FLE for connecting the color camera CCm and the IR camera DCm, an interface CTa of the color camera CCm, and an interface CTb of the IR camera DCm.

[0033] Meanwhile, the connection member FLE for connecting the color camera CCm and the IR camera DCm is flexible. That is, unlike FIG. 1A, the bracket BRK may not be provided. Accordingly, having no bracket, the camera device 100 or the electronic device in which the camera device 100 is mounted may have an improved degree of freedom of design.

[0034] Meanwhile, in the camera device 100 of FIG. 1B, a distance between the color camera CCm and the IR camera DCm and the like may change irregularly, in which case calibration is required between a color image and an IR image. The calibration between the color image and the IR image will be described with reference to FIG. 6 and the following figures.

[0035] Meanwhile, the camera device 100 according to the embodiment of FIG. 1B of the present disclosure may be provided in various electronic devices.

[0036] FIG. 2 is a diagram explaining various examples of an electronic device.

[0037] Referring to FIG. 2, the camera device 100 according to the embodiment of FIG. 1B of the present disclosure may be employed in a mobile terminal 200, an air conditioner 200a, a robot cleaner 200b, a refrigerator 200c, a washing machine 200d, a TV 200c, a vehicle, a drone, and the like.

[0038] The following description will be made based on the camera device 100 provided in the mobile terminal 200.

[0039] FIG. 3A is an internal cross-sectional view of a color camera and an IR camera of FIG. 1B.

[0040] Referring to the drawing, the color camera CCm may include an aperture 194a, a lens device 193a, and an image sensor Imsa.

[0041] The aperture 194a may permit or prevent light incident upon the lens device 193a.

[0042] The lens device 193a may include a plurality of lenses which are adjusted for focus variation.

[0043] For sensing RGB colors, the image sensor Imsa may include a RGb filter 915a, and a sensor array 911a for converting an optical signal into an electrical signal.

[0044] Accordingly, the image sensor Imsa may sense and output a color image.

[0045] The IR camera DCm may include an aperture 194b, a lens device 193b, and an image sensor Imsb.

[0046] The aperture 194b may permit or prevent light incident upon the lens device 193b.

[0047] The lens device 193b may include a plurality of lenses which are adjusted for focus variation.

[0048] For sensing an IR image, the image sensor Imsb may include an IR filter 915b, and a sensor array 911a for converting an optical signal into an electrical signal.

[0049] Accordingly, the image sensor Imsb may sense and output an IR image.

[0050] Meanwhile, in addition to the color image, the color camera CCm may output images of various colors. For example, the color camera CCm may output a W color image, RGBY images, or the like.

[0051] FIG. 3B is an internal block diagram of a camera device including the color camera and the IR camera of FIG. 1B.

[0052] Referring to the drawing, the camera device 100 includes the color camera CCm, the IR camera DCm, a processor 170, a sensor device 130, a memory 140, a power supply 190, and an interface 150.

[0053] For outputting a color image, the color camera CCm may include the lens device 193a and the image sensor Imsa.

[0054] The lens device 193a in the color camera CCm may include a plurality of lenses which receive the incident light and are adjusted for focus variation.

[0055] Meanwhile, an exposure time of the image sensor Imsa may be adjusted based on an electronic signal.

[0056] For outputting an IR image, the IR camera DCm may include the lens device 193b and the image sensor Imsb.

[0057] The lens device 193b in the IR camera DCm may include a plurality of lenses which receive the incident light and are adjusted for focus variation.

[0058] Meanwhile, the processor 170 may receive the color image which is based on the electronic signal from the image sensor Imsa in the color camera CCm. Alternatively, the processor 170 may generate a color image based on the electronic signal from the image sensor Imsa in the color camera CCm.

[0059] Meanwhile, the processor 170 may receive the IR image which is based on the electronic signal from the image sensor Imsb in the IR camera DCm. Alternatively, the processor 170 may generate an IR image based on the electronic signal from the image sensor Imsb in the IR camera DCm.

[0060] Meanwhile, the processor 170 may calculate error information based on a difference between the color image, obtained from the color camera CCm, and the IR image from the IR camera DCm, and may compensate for at least one of the color image and the IR image based on the calculated error information and may output the compensated color image or the compensated IR image.

[0061] Meanwhile, by analyzing each feature point of the color image from the color camera CCm and the IR image from the IR camera, the processor 170 may calculate three-dimensional (3D) error information, and may compensate for at least one of the color image and the IR image based on the calculated 3D error information, to output the compensated color image or the compensated IR image.

[0062] Meanwhile, the processor 170 may upscale the IR image, may calculate 3D error information between the upscaled IR image and the color image, and may compensate for at least one of the color image and the IR image based on the calculated 3D error information, to output the compensated color image or the compensated IR image.

[0063] Meanwhile, by matching the compensated and upscaled IR image with the color image based on the calculated 3D error information, the processor 170 may output the compensated color image.

[0064] Meanwhile, by comparing a luminance component of a color image Imgr with a luminance component of the IR image Imgt, the processor 170 may calculate error information, and may compensate for at least one of the color image and the IR image based on the calculated error information to output the compensated color image or the compensated IR image.

[0065] The sensor device 130 may sense movement information, position information, or the like of the camera device 100. To this end, the sensor device 130 may include a GPS receiving unit, an inertial sensor (gyro sensor, acceleration sensor, etc.), and the like.

[0066] The memory 140 may store data for operation of the camera device 100 or the color image from the color camera CCm, the IR image from the IR camera DCm, or the compensated color image or the compensated IR image.

[0067] The interface 150 may be used for data transmission with other units of the camera device 100.

[0068] The power supply 190 may supply power for operation of the camera device 100.

[0069] For example, the power supply 190 may convert DC power or AC power input from an external source and may supply the converted DC power to the processor 170, the color camera CCm, the IR camera DCm, the sensor device 130, the memory 140, the interface 150, and the like.

[0070] FIGS. 4A to 5C are diagrams referred to in the description of an operation of the camera device of FIG. 3B.

[0071] First, (a) of FIG. 4A illustrates an example of calibrating an IR image Rim due to a distortion between a color image Lim obtained from the color camera CCm and an IR image Rim obtained from the IR camera DCm. Specifically, (a) of FIG. 4A illustrates an IR image R'im obtained by compensating for the IR image Rim.

[0072] The compensated IR image Rim may be an image which is compensated based on at least one of 3D translation, 3D shift, and 3D rotation.

[0073] Here, the 3D rotation may include yaw, pitch, and roll rotations.

[0074] Then, (b) of FIG. 4A illustrates the color image Lim and the compensated IR image R'im which are aligned regularly. Accordingly, by matching the color image Lim with the compensated IR image R'im, the compensated color image or the compensated IR image may be output.

[0075] FIG. 4B illustrates an example of photographing the same pattern PAT by using the color camera CCm and the IR camera DCm.

[0076] A misalignment, such as [R1, T1. , occurs due to the 3D rotation, 3D translation, and the like of the color camera CCm, such that a color image img1 is obtained from the color camera CCm.

[0077] Then, a misalignment, such as [R2, T2. , occurs due to the 3D rotation, 3D translation, and the like of the IR camera DCm, such that an IR image img2 is obtained from the IR camera DCm.

[0078] Meanwhile, a difference between the color image img1 and the IR image img2 may be represented by a degree of distortion H.

[0079] Meanwhile, in order to minimize a difference between the color image img1 and the IR image img2, the processor 170 may calculate a degree of distortion H, which is a difference between the color image img1 and the IR image img2, misalignment information [R1, T1. , such as rotation, translation, and the like of the color image img1, and misalignment information [R2, T2. , such as rotation, translation, and the like of the IR image img2.

[0080] Further, based on the degree of distortion H, the misalignment information [R1, T1. , such as rotation, translation, and the like of the color image img1, and the misalignment information [R2, T2. , such as rotation, translation, and the like of the IR image img2, the processor 170 may output the compensated color image or the compensated IR image for reducing an error.

[0081] FIG. 5A illustrates a color image 510 obtained from the color camera CCm and an IR image 515 obtained from the IR camera DCm in the camera device 100.

[0082] If there is a difference in resolution between the color camera CCm and the IR camera DCm, a resolution of the color image 510 may be higher than a resolution of the IR image 515.

[0083] First, in order to perform calibration based on the difference between the color image 510 and the IR image 515, it is required to adjust the resolution of the color image 510 to be the same as the resolution of the IR image 515.

[0084] To this end, the processor 170 may upscale the IR image.

[0085] Alternatively, the processor 170 may upscale the IR image and may downscale the color image.

[0086] FIG. 5B illustrates a downscaled color image 520 and an upscaled IR image 525.

[0087] Meanwhile, when upscaling a low-resolution IR image, a blur and the like may occur according to a pattern included in the IR image.

[0088] FIG. 5C illustrates an IR image IArm captured of an area Arm in a black-and-white square grid pattern.

[0089] In the IR image IArm, a blur may occur in boundaries EGa and EGb between black and white areas, and this phenomenon is more noticeable when the IR image is upscaled.

[0090] Accordingly, a pattern used for calibration of the color camera CCm and the IR camera DCm is preferably a circular grid pattern as illustrated in FIG. 7, rather than a square grid pattern. Meanwhile, in order to determine a reference point and the like, a grid pattern of circles of a plurality of sizes may be preferable.

[0091] FIG. 6 is a flowchart illustrating an operating method of a camera device, and FIGS. 7 to 10F are diagrams referred to in the description of the operating method of FIG. 6.

[0092] First, referring to FIG. 6, the processor 170 in the camera device 100 obtains a color image from the color camera CCm (S610).

[0093] Then, the processor 170 in the camera device 100 obtains an IR image from the IR camera DCm (S615).

[0094] The obtained color image and IR image may be images captured of the same pattern or object.

[0095] For example, a color image and an IR image captured of a grid pattern PAT of circles of a plurality of sizes as illustrated in FIG. 7 may be obtained at a first time point.

[0096] In the grid pattern PAT of circles of a plurality of sizes illustrated in FIG. 7, circular patterns Ba and Bb of a first size are disposed at the center thereof, and circular patterns Bc and Bd of a second size are disposed around the circular patterns Ba and Bb of the first size.

[0097] In this case, the circular patterns may be in black or in colors, and a background color around the circular patterns may be white.

[0098] FIG. 8A illustrates a top view of the color camera CCm and the IR camera DCm capturing an image of the grid pattern PAT of circles of a plurality of sizes, and FIG. 8B illustrates a side view of the color camera CCm and the IR camera DCm capturing an image of the grid pattern PAT of circles of a plurality of sizes.

[0099] Referring to FIGS. 8A and 8B, the color camera CCm and the IR camera DCm are spaced apart from each other by a predetermined distance, in which the color camera CCm may acquire a color image Imgr, and the IR camera DCm may acquire an IR image Imgt, of the grid pattern PAT of circles of a plurality of sizes.

[0100] As described above, if there is a difference in resolution between the color camera CCm and the IR camera DCm, the processor 170 performs upscaling and the like of the IR image Imgt.

[0101] Then, the processor 170 in the camera device 100 may calculate 3D error information by analyzing each feature point of the color image and the IR image (S670).

[0102] Subsequently, the processor 170 in the camera device 100 may output a compensated color image or a compensated IR image based on the calculated 3D error information (S680).

[0103] Meanwhile, the processor 170 may calculate error information based on the color image Imgr obtained from the color camera CCm and the IR image Imgt obtained from the IR camera DCm, and may compensate for at least one of the color image Imgr and the IR image Imgt based on the calculated error information to output the compensated color image or the compensated IR image. Accordingly, a difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0104] Meanwhile, by analyzing each feature point of the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm, the processor 170 may calculate 3D error information, and may compensate for at least one of the color image Imgr and the IR image Imgt based on the calculated 3D error information to output the compensated color image 540 or the compensated IR image Imgt. Particularly, based on the calculated 3D error information, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0105] Meanwhile, the processor 170 may upscale the IR image Imgt, may calculate 3D error information between the upscaled IR image Imgt and the color image Imgr, and may compensate for at least one of the color image Imgr and the IR image Imgt based on the calculated 3D error information to output the compensated color image or the compensated IR image. Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0106] Meanwhile, the 3D error information may include relative rotation information, relative translation information, or relative shift information between the color image Imgr and the IR image Imgt. Accordingly, based on the relative rotation information, relative translation information, or relative shift information, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0107] Meanwhile, by matching the compensated and upscaled IR image Imgt with the color image Imgr based on the calculated 3D error information, the processor 170 may output a compensated color image. Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0108] Meanwhile, by comparing a luminance component of the color image Imgr with a luminance component of the IR image Imgt, the processor 170 may calculate error information, and may compensate for at least one of the color image Imgr and the IR image Imgt based on the calculated error information to output the compensated color image or the compensated IR image. Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0109] FIG. 9 illustrates an example of an internal block diagram of the processor 170.

[0110] Referring to the drawing, the processor 170 includes a feature point analyzer 910 configured tc analyze each feature point of the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm; an external variable calculator 920 configured to calculate an external variable based on each feature point information analyzed by the feature point analyzer 910; an error calculator 930 configured to calculate 3D error information based on the calculated external variable; and an error corrector 940 configured to correct an error based on the 3D error information. Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0111] FIG. 10A illustrates the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm, which are input to the feature point analyzer 910.

[0112] Meanwhile, the feature point analyzer 910 may detect a pattern or a reference point from each of the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm, and may calculate a direction vector based on the detected pattern or reference point.

[0113] Meanwhile, the feature point analyzer 910 may detect a circular pattern from the color image Imgr and the IR image Imgt, and may detect a reference point in the circular pattern.

[0114] Meanwhile, the feature point analyzer 910 may calculate a direction vector based on the detected circular pattern or reference point.

[0115] Further, the feature point analyzer 910 may analyze the feature point based on the detected circular pattern or reference point, or the direction vector. That is, the feature point may include the circular pattern, the reference point, or the direction vector.

[0116] Meanwhile, based on each feature point information analyzed by the feature point analyzer 910, the external variable calculator 920 may calculate rotation information, translation information, or shift information of each of the color image Imgr and the IR image Imgt.

[0117] Meanwhile, the external variable calculator 920 calculates a homography between the color image Imgr and the IR image Imgt.

[0118] Meanwhile, based on the calculated homogrpahy, the external variable calculator 920 may calculate the rotation information, the translation information, or the shift information of aech of the color image Imgr and the IR image Imgt.

[0119] Meanwhile, based on the rotation information, the translation information, or the shift information of each of the color image Imgr and the IR image Imgt, the error calculator 930 may calculate relative rotation information, relative translation information, or relative shift information between the color image Imgr and the IR image Imgt.

[0120] Meanwhile, the error calculator 930 may calculate a reprojection error based on the color image Imgr and the IR image Imgt.

[0121] FIG. 10B illustrates an example of calculating a reprojection error based on the color image Imgr and a compensated IR image I'mgt.

[0122] The reprojection error information may include the above relative rotation information, relative translation information, or relative shift information.

[0123] Meanwhile, by matching the compensated and upscaled IR image Imgt with the color image Imgr based on the calculated 3D error information, the error corrector 940 may output a compensated color image. Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0124] Meanwhile, the error corrector 940 may output the compensated color image or the compensated IR image based on the calculated 3D error information.

[0125] FIG. 10C illustrates an example of a compensated IR image Imgev.

[0126] Accordingly, the difference between the color image Imgr from the color camera CCm and the IR image Imgt from the IR camera DCm may be compensated rapidly and accurately.

[0127] FIG. 10D illustrates an example of another color image and another IR image.

[0128] Referring to the drawing, the processor 170 may receive a color image 510 from the color camera CCm of the camera device 100 and an IR image 515 from the IR camera DCm.

[0129] If there is a difference in resolution between the color camera CCm and the IR camera DCm, a resolution of the color image 510 may be higher than a resolution of the IR image 515, as illustrated in the drawing.

[0130] Accordingly, the processor 170 may downscale the color image 510 and may upscale the IR image 515.

[0131] FIG. 10E illustrates a downscaled color image 520 and an upscaled IR image 525.

[0132] Meanwhile, the processor 170 may perform matching by using the downscaled color image 520 and the upscaled IR image 525.

[0133] For example, the processor 170 may match the upscaled IR image 525 to the downscaled color image 520. Further, the processor 170 may calculate error information based on the difference.

[0134] For example, by analyzing each feature point based on the downscaled color image 520 and the upscaled IR image 525, the processor 170 may calculate 3D error information.

[0135] Specifically, by analyzing each feature point based on the downscaled color image 520 and the upscaled IR image 525, the processor 170 may calculate 3D error information, such as the relative rotation information, relative translation information, or relative shift information, between the color image Imgr and the IR image Imgt.

[0136] In addition, the processor 150 may output the compensated color image or the compensated IR image based on the calculated 3D error information.

[0137] FIG. 10F illustrates an example of a compensated color image 540. Unlike the drawing, a compensated IR image may also be output.

[0138] Accordingly, the difference between the color image from the color camera and the IR image from the IR camera may be compensated rapidly and accurately.

[0139] FIGS. 11A to 11C are diagrams referred to in the description of misalignment occurring between a color camera and an IR camera in the case where there is an impact on a camera device.

[0140] Referring to FIG. 11A, if an impact is applied to a side surface of the camera device 100, misalignment occurs between the color camera CCm and the IR camera DCm.

[0141] Accordingly, if there is no impact as illustrated in FIG. 11B, it is not appropriate to use 3D information as it is, which is calculated based on a difference between a color image 1110 from the color camera CCm and an IR image 1115 from the IR camera DCm.

[0142] FIG. 11B illustrates a synthesized image 1120 using the color image 1110 from the color camera CCm and the IR image 1115 from the IR camera DCm, in the case where there is no impact.

[0143] Meanwhile, of a color image 1130 from the color camera CCm and an IR image 1135 from the IR camera DCm, at least the IR image 1135 is input while being misaligned when there is no impact as illustrated in FIG. 11C.

[0144] Accordingly, if the processor 170 generates a synthesized image by using pre-stored 3D information as it is, a synthesized image 1140 having an error is generated as illustrated in FIG. 11C.

[0145] A method of compensating for misalignment or distortion, caused by an external impact, movement of the camera device 100, etc., between the color camera CCm and the IR camera DCm will be described with reference to FIG. 12 and the following figures.

[0146] FIG. 12 is a flowchart illustrating an operating method of a camera device according to an embodiment of the present disclosure.

[0147] Referring to the drawing, the processor 170 in the camera device 100 obtains a color image from the color camera CCm (S1110).

[0148] The, the processor 170 in the camera device 100 obtains an IR image from the IR camera DCm (S1115).

[0149] The obtained color image and IR image may be images captured of the same pattern or object.

[0150] For example, a color image and an IR image captured of the grid pattern PAT of circles of a plurality of sizes as illustrated in FIG. 7 may be obtained at a first time point.

[0151] In another example, a color image and an IR image including a common face area may be obtained as illustrated in FIG. 13A.

[0152] Then, based on a signal input by the sensor device 130, the processor 170 in the camera device 100 may determine whether an impact is sensed or whether the sensed impact is greater than or equal to a first level (S1120).

[0153] Further, if the sensed impact is greater than or equal to the first level, the processor 170 in the camera device 100 may extract a first region of the color image and may extract a second region of the IR image (S1125).

[0154] The first region and the second region may include a face region.

[0155] That is, if the impact sensed by the sensor device 130 is greater than or equal to the first level, the processor 170 in the camera device 100 may extract the face region from each of the color image and the IR image.

[0156] Subsequently, the processor 170 in the camera device 100 may extract a gradient of the extracted face region (S1130).

[0157] The gradient may include a gradient of rotation information, translation information, or shift information of the color image 1130 or the IR image 1135.

[0158] Next, the processor 170 in the camera device 100 may compensate for 3D error information based on the extracted gradient (S1170).

[0159] Then, the processor 170 in the camera device 100 may output a compensated color image or a compensated depth image based on the compensated 3D error information (S1180).

[0160] Meanwhile, the processor 170 upscales the IR image 1135, calculates 3D error information based on a difference between a gradient of the color image and a gradient of the upscaled IR image, and may compensate for at least one of the color image 1130 an the IR image 1135 based on the calculated 3D error information to output the compensated color image 1350 or the compensated IR image. Particularly, based on the calculated 3D error information, a difference between the color image 1130 from the color camera CCm and the IR image 1135 from the IR camera DCm may be compensated rapidly and accurately.

[0161] FIG. 13A illustrates the color image 1130 from the color camera CCm and the IR image 1135 from the IR camera DCm.

[0162] If there is no difference in resolution between the color camera CCm and the IR camera DCm, a resolution of the color image 1130 may be higher than a resolution of the IR image 1135.

[0163] Meanwhile, the processor 170 upscales the IR image 1135, and may extract a face region 1320 from the upscaled IR image as illustrated in FIG. 13B, to calculate a gradient of the extracted face region 1320.

[0164] Meanwhile, the processor 170 may extract a face region 1310 from the color image 1130 as illustrated in FIG. 13B and may calculate a gradient of the extracted face region 1310.

[0165] In FIG. 13B, (b) illustrates a gradient 1315 related to the face region of the color image.

[0166] Meanwhile, based on a difference between a gradient of the color image and a gradient of the IR image, the processor 170 may calculate 3D error information, and may compensate for at least one of the color image 1130 and the IR image 1135 based on the calculated 3D error information to output the compensated color image 1350 or the compensated IR image. Particularly, based on the calculated 3D error information, a difference between the color image 1130 from the color camera CCm and the IR image 1135 from the IR camera DCm may be compensated rapidly and accurately.

[0167] Meanwhile, the processor 170 normalizes the respective gradients of the color image and the IR image, calculates 3D error information based on a difference between the normalized gradients, and may compensate for least one of the color image 1130 and the IR image 1135 based on the calculated 3D error information to output the compensated color image 1350 or the compensated IR image. Particularly, based on the normalization, a difference between the color image 1130 from the color camera CCm and the IR image 1135 from the IR camera DCm may be compensated rapidly and accurately.

[0168] FIG. 13C illustrates an example of calculating dx and dy by normalizing the respective gradients 1315 and 1320 of the color image and the IR image.

[0169] Next, FIG. 13D illustrates an example of calculating dθ and recalculating dx and dy based on the normalization of FIG. 13C.

[0170] Meanwhile, the processor 170 may calculate a distortion degree H of homography based on the calculated dθ and the recalculated dx and dy, and may compensate for 3D translation information, 3D shift information, 3D rotation information, and the like based on the calculation.

[0171] For example, the processor 170 updates 3D error information, which is calculated and pre-stored as illustrated in FIG. 6 and the like, by using the calculated compensation value based on FIGS. 13B to 13D.

[0172] Further, by compensating for at least one of the color image 1130 and the IR image 1135 based on the updated 3D error information, the processor 170 may output the compensated color image 1350 or the compensated IR image.

[0173] Meanwhile, the processor 170 may extract a first region 1310 from the color image 1130 obtained from the color camera CCm, may extract a second region 1320 from the IR image 1135 obtained from the IR camera DCm, may calculate error information based on a difference between a gradient of a luminance component of the first region 1310 and a gradient of a luminance component of the second region 1320, and may compensate for at least one of the color image 1130 and the IR image 1135 based on the calculated error information to output the compensated color image 1350 or the compensated IR image. Accordingly, a difference between the color image 1130 from the color camera CCm and the IR image 1135 from the IR camera DCm may be compensated rapidly and accurately.

[0174] FIG. 13E illustrates an example in which under external impact, an IR image and a color image, which are uncompensated, are synthesized and misaligned.

[0175] Referring to the drawing, face regions and the like do not coincide in a synthesized image 1340.

[0176] FIG. 13F illustrates an example in which under external impact, the IR image and the color image, which are compensated, are synthesized and aligned.

[0177] Referring to the drawing, face regions and the like coincide in a synthesized image 135. Accordingly, a difference between the color image and the IR image may be compensated rapidly and accurately.

Claims

1. A camera device (100) comprising: a color camera (CCm); an IR camera (DCm); a flexible connection member (FLE) connecting the color camera (CCm) and the IR camera (DCm); a sensor device (130) for sensing movement or impact; and a processor (170) comprising: a feature point analyzer (910) configured to analyze each feature point of the color image (Imgr) from a color camera (CCm) and an IR image (Imgt) from the IR camera (DCm); an external variable calculator (920) is configured to calculate rotation information, translation information, or shift information of each of the color image (Imgr) and the IR image (Imgt), based on the each feature point information analyzed by the feature point analyzer (910); an error calculator (930) configured to calculate 3D error information including relative rotation information, relative translation information, or relative shift information between the color image (Imgr) and the IR image (Imgt), wherein the relative rotation information, the relative translation information, or the relative shift information between the color image (Imgr) and the IR image (Imgt) is calculated based on the rotation information, the translation information, or the shift information of each of the color image (Imgr) and the IR image (Imgt); and an error corrector (940) configured to correct an error based on the 3D error information, wherein the processor (170) is configured to: calculate a homography between the color image (Imgr) and the IR image (Imgt) based on the color image (Imgr) from the color camera (CCm) and the IR image (Imgt) from the IR camera (DCm), calculate the 3D error information based on the calculated homogrpahy, compensate for at least one of the color image (Imgr) and the IR image (Imgt) based on the calculated 3D error information, and output a compensated color image or a compensated IR image, wherein the calculated 3D error information is stored in the camera device (100), wherein, in response to the movement or the impact sensed by the sensor device (130) being greater than or equal to a first level, the processor is further configured to: upscale the IR image (Imgt), in response to a resolution of the color image (Imgr) being higher than a resolution of the IR image (Imgt), extract a first region of the color image (Imgr) from the color camera (CCm), extract a second region of the IR upscaled image (Imgt) from the IR camera (DCm), normalize a gradient of the first region and a gradient of the second region; compensate and update the stored 3D error information based on a difference between the normalized gradient of the first region and the normalized gradient of the second region, compensate for at least one of the color image (Imgr) and the IR image (Imgt) based on the compensated and updated 3D error information, and output a compensated color image or a compensated IR image (Imgt).

2. The camera device (100) of claim 1, wherein the first region and the second region comprise a face region.

3. The camera device (100) of claim 1, wherein the gradient comprises gradients of rotation information, translation information, or shift information of the color image (Imgr) or the IR image (Imgt).

4. The camera device (100) of claim 1, wherein, in response to the movement or the impact sensed by the sensor device (130) being greater than or equal to the first level, the processor (170) is configured to compensate the 3D error information based on a difference between a gradient of a luminance component of the first region and a gradient of a luminance component of the second region.

5. An electronic device comprising the camera device (100) of any one of claims 1 to 4.