Apparatus for correcting depth image and method thereof

The depth image correction device addresses inaccuracies in ToF depth images by using feature point detection and correction coefficients derived from machine learning, enhancing accuracy and object representation in 3D imaging.

KR103044571B1Active Publication Date: 2026-11-06LG INNOTEK CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
KR1020200119803
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-17
Publication Date
2026-11-06
Estimated Expiration
2040-09-17

AI Technical Summary

Technical Problem

Existing depth image acquisition methods, particularly the Time of Flight (ToF) method, face challenges in accurately measuring depth information due to variations in object shape and material, leading to distorted or inaccurate depth images.

Method used

A depth image correction device and method that utilizes a detection unit to identify feature points, selects correction coefficients from a table based on object pose, distance, and shape information, and applies these coefficients to correct depth and color information using machine learning algorithms to generate accurate depth images.

Benefits of technology

The solution effectively corrects depth image distortions, improving accuracy and providing additional object type information, ensuring precise representation of objects in 3D space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112020098844391-PAT00009_ABST
    Figure 112020098844391-PAT00009_ABST
Patent Text Reader

Abstract

A depth image correction device according to an embodiment of the present invention comprises: a detection unit that detects a feature point corresponding to an object from a first image, which is an IR image of an object; a selection unit that selects a first correction coefficient of shape information corresponding to the feature point from a table in which a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information is stored; a calculation unit that calculates a distance error for the object based on the first correction coefficient; and a depth correction unit that applies the distance error to a second image in which depth information for the object is measured to correct the depth information of the second image.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The embodiment relates to a depth image correction device and a method thereof. Background Technology

[0002] 3D content is being applied in many fields, including not only games and culture but also education, manufacturing, and autonomous driving, and depth information (Depth Map) is required to acquire 3D content. Depth information is data representing spatial distance and indicates the perspective information of one point in a 2D image relative to another point.

[0003] Methods used to acquire depth information include projecting Infrared (IR) structured light onto an object, utilizing stereo cameras, and the Time of Flight (TOF) method. According to the TOF method, the distance to an object is calculated by measuring the time of flight—that is, the time it takes for light to be emitted and reflected back. The greatest advantage of the TOF method is that it provides distance information in 3D space rapidly in real time. Furthermore, users can obtain accurate distance information without applying separate algorithms or hardware corrections. It can also acquire accurate depth information even when measuring very close or moving subjects.

[0004] However, in the case of the ToF method, there is a problem in that depth information is not measured or inaccurate depth information is generated depending on the shape or material of the object. The problem to be solved

[0005] The embodiment is intended to provide a depth image correction device and a method for improving image quality by correcting distortion of a depth image.

[0006] The problem to be solved in the embodiments is not limited thereto, and may also include objectives or effects that can be identified from the means of solving the problem or the forms of implementation described below. means of solving the problem

[0007] A depth image correction device according to an embodiment of the present invention comprises: a detection unit that detects a feature point corresponding to an object from a first image, which is an IR image of an object; a selection unit that selects a first correction coefficient of shape information corresponding to the feature point from a table in which a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information is stored; a calculation unit that calculates a distance error for the object based on the first correction coefficient; and a depth correction unit that applies the distance error to a second image in which depth information for the object is measured to correct the depth information of the second image.

[0008] The above table can be generated through a machine learning algorithm that generates the first correction coefficient using the boundary information of the object and the depth information of the object detected from at least one of the IR image or RGB image taken of the object at a previous time point as training data.

[0009] The above training data can be generated using IR images or RGB images captured at multiple poses and multiple distances for each of multiple types of objects.

[0010] It may include a feature point correction unit that detects color information corresponding to the feature point from a third image, which is an RGB image of the object, and corrects the feature point based on the color information.

[0011] The above feature point correction unit can select a second correction coefficient corresponding to the detected color information from the table in which a second correction coefficient corresponding to the color reflectance is stored, correct a pixel value corresponding to the feature point based on the selected second correction coefficient, and correct the feature point based on the corrected pixel value of the feature point.

[0012] It may include an output unit that selects and provides an object type corresponding to the feature point from the above table.

[0013] A depth image correction method according to an embodiment of the present invention comprises: a step of detecting a feature point corresponding to an object from a first image, which is an IR image of an object; a step of selecting a first correction coefficient of shape information corresponding to the feature point from a table in which first correction coefficients corresponding to the pose of the object, the distance between the camera and the object, and shape information of the object are stored; a step of calculating a distance error for the object based on the first correction coefficient; and a step of correcting the depth information of the second image by applying the distance error to a second image in which depth information for the object is measured.

[0014] The above table can be generated through a machine learning algorithm that generates the first correction coefficient using the boundary information of the object and the depth information of the object detected from at least one of the IR image or RGB image taken of the object at a previous time point as training data.

[0015] The above training data can be generated using IR images or RGB images captured at multiple poses and multiple distances for each of multiple types of objects.

[0016] The method may include: a step of detecting color information corresponding to the feature point from a third image, which is an RGB image of the object captured above; and a step of correcting the feature point based on the color information.

[0017] The step of correcting the feature point may include: selecting a second correction coefficient corresponding to the color information detected from the table in which a second correction coefficient corresponding to the color reflectance is stored; correcting a pixel value corresponding to the feature point based on the selected second correction coefficient; and correcting the feature point based on the corrected pixel value for the feature point.

[0018] It may include a step of providing an object type corresponding to the feature point from a previously stored table. Effects of the invention

[0019] According to an embodiment, depth distortion can be corrected to improve the accuracy of depth information for an object.

[0020] It can provide additional information about the object, such as the type of the object.

[0021] The various and beneficial advantages and effects of the present invention are not limited to those described above and may be more easily understood in the process of explaining specific embodiments of the present invention. Brief explanation of the drawing

[0022] FIG. 1 is a drawing showing a camera device according to one embodiment of the present invention. FIG. 2 is a drawing showing a camera device according to another embodiment of the present invention. FIG. 3 is a configuration diagram of a depth image correction device according to an embodiment of the present invention. FIG. 4 is a drawing for explaining a table according to an embodiment of the present invention. FIG. 5 is a flowchart of a depth image correction method according to a first embodiment of the present invention. FIG. 6 is a flowchart of a depth image correction method according to a second embodiment of the present invention. Figure 7 is a simulation result according to an embodiment of the present invention. Specific details for implementing the invention

[0023] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0024] However, the technical concept of the present invention is not limited to some of the described embodiments but can be implemented in various different forms, and within the scope of the technical concept of the present invention, one or more of the components among the embodiments may be selectively combined or substituted.

[0025] In addition, terms used in the embodiments of the present invention (including technical and scientific terms) may be interpreted in a meaning that is generally understood by those skilled in the art to which the present invention belongs, unless explicitly and specifically defined otherwise. Terms that are commonly used, such as terms defined in advance, may be interpreted in consideration of their meaning in the context of the relevant technology.

[0026] Furthermore, the terms used in the embodiments of the present invention are for the purpose of describing the embodiments and are not intended to limit the present invention.

[0027] In this specification, the singular form may include the plural form unless specifically stated otherwise in the text, and when described as "at least one of A and B and C (or more than one)," it may include one or more of all combinations that can be formed from A, B, and C.

[0028] In addition, terms such as first, second, A, B, (a), (b), etc. may be used when describing the components of the embodiments of the present invention.

[0029] These terms are intended merely to distinguish a component from other components and are not limited by the nature, order, sequence, etc., of the said component.

[0030] And, where it is stated that a component is 'connected', 'combined', or 'joined' to another component, this may include not only cases where the component is directly connected, combined, or joined to the other component, but also cases where it is 'connected', 'combined', or 'joined' due to another component located between the component and the other component.

[0031] Furthermore, when described as being formed or placed "above or below" each component, "above" or "below" includes not only cases where two components are in direct contact with each other, but also cases where one or more other components are formed or placed between the two components. Additionally, when expressed as "above or below," it may include the meaning of a downward direction as well as an upward direction relative to a single component.

[0032] FIG. 1 is a drawing showing a camera device according to one embodiment of the present invention.

[0033] Referring to FIG. 1, a camera device (1) according to an embodiment of the present invention may include a ToF camera module (10). The camera device (1) may refer to a terminal device equipped with a ToF camera module (10). For example, the camera device (1) may be a mobile terminal equipped with a ToF camera module (10).

[0034] A ToF camera module (10) may refer to a camera module that generates IR images and depth information by irradiating light onto an object and receiving light reflected from the object. To this end, the ToF camera module (10) may include a light-emitting unit (11), a light-receiving unit (12), and a processing unit (13). The light-emitting unit (11), the light-receiving unit (12), and the processing unit (13) may be arranged on a single substrate and housed within a single housing to form a single module.

[0035] The light-emitting unit (11) can generate light through a light source and irradiate it onto an object. To this end, the light-emitting unit (11) may include a light source and an optical member.

[0036] The light-emitting unit (11) may include a light source disposed on a circuit board. The light source generates and emits light. The light source may emit light to an upper surface facing a lower surface in contact with the circuit board. The light source has a structure in which a plurality of layers are stacked, and when current is applied, light may be generated from the plurality of layers. The generated light may pass through an aperture inside the light source and then be output through a plurality of openings disposed on the upper surface. There may be multiple apertures, and each aperture may correspond to each of the plurality of openings. Therefore, the number of apertures and the number of openings may be the same. According to an embodiment, the light source may be a vertical-cavity surface-emitting laser (VCSEL) array.

[0037] The light-emitting unit (11) may include an optical member positioned at the top of the light source. The optical member may scatter and emit light emitted from the light source. In this case, the optical member may be a diffuser. The optical member may be implemented in the form of a plate including a first surface and a second surface. The first surface of the optical member may be a surface that receives light output from the light source. Micro lenses are arranged on the first surface of the optical member according to a certain pitch. In this case, the angle of light collected through the first surface is adjusted according to the size, curvature, refractive index, and pitch size of the micro lenses, thereby scattering the input light and outputting it through the second surface. The size, curvature, refractive index, and pitch size of the micro lenses may be modified by a person skilled in the art, taking into account the use of the camera module according to the embodiment of the present invention, the distance between the optical member module and the light source, and the shape and type of the light source.

[0038] The light receiving unit (12) can collect light reflected from an object and convert it into an electrical signal. To this end, the light receiving unit (12) may include a lens, a filter, and a sensor.

[0039] The lens can concentrate the received light. The lens can be fixed within a barrel. The lens can be a plastic injection molded product. The lens can be formed by plastic injection molding. The lens may include a plurality of lenses.

[0040] The filter can be a band-pass filter that allows light of a specific wavelength range to pass through. The filter can pass infrared light. The filter can pass light of specific wavelengths within the infrared spectrum. The filter can pass light within the wavelength band emitted by a light source. The filter can block visible light. The filter can be placed between the lens and the sensor.

[0041] The sensor can be placed on a circuit board. The sensor can detect infrared light. The sensor can detect light of a specific wavelength among infrared light. The sensor can detect light that has passed through a filter. The sensor can detect light within the wavelength band of a light source. In one embodiment, the sensor may be a Time of Flight (ToF) sensor. A ToF sensor can sense 3D image information of a subject by detecting light emitted from a light source and reflected off the subject.

[0042] The processing unit (13) can generate an image based on an electrical signal received from the light receiving unit (12). The processing unit (13) receives an electrical signal from the sensor of the light receiving unit (12) and can generate an image based on the electrical signal.

[0043] The computation unit (13) can generate a three-dimensional depth image based on a signal received from the light receiving unit (12). According to one embodiment, the computation unit (13) can generate a depth image using the following mathematical formulas 1 and 2. The computation unit (13) can generate a depth image using the following mathematical formula 1 to determine the phase difference (t) between the output light signal (light output from the light emitting unit (11)) and the input light signal (light input to the light receiving unit (12)). d ) can be calculated.

[0044]

[0045] Here, Q1 to Q4 are the charge amounts of each of the four electrical signals. Q1 is the charge amount of the electrical signal corresponding to the reference signal having the same phase as the output light signal. Q2 is the charge amount of the electrical signal corresponding to the reference signal having a phase 180 degrees behind the output light signal. Q3 is the charge amount of the electrical signal corresponding to the reference signal having a phase 90 degrees behind the output light signal. Q4 is the charge amount of the electrical signal corresponding to the reference signal having a phase 270 degrees behind the output light signal.

[0046] In addition, the processing unit can calculate the distance between the object and the camera device (1) using the phase difference between the output light signal and the input light signal. At this time, the processing unit can calculate the distance (d) between the object and the camera device (1) using the following mathematical formula 2.

[0047]

[0048] Here, c is the speed of light and f is the frequency of the output light.

[0049] The computation unit (13) can generate a two-dimensional IR image based on a signal received from the light receiving unit (12). According to one embodiment, if the processing unit performs an operation as in Equation 3 using four phase images, it can obtain an amplitude image, which is a ToF IR image.

[0050]

[0051] Here, Raw(x0) is a pixel-wise data value received by the sensor at phase 0°, Raw(x90) is a pixel-wise data value received by the sensor at phase 90°, Raw(x180) is a pixel-wise data value received by the sensor at phase 180°, and Raw(x270) is a pixel-wise data value received by the sensor at phase 270°.

[0052] In another embodiment, by performing calculations as in Equation 4 using the four phase images of FIG. 6, the processing unit may obtain an intensity image, which is another ToF IR image.

[0053]

[0054] Here, Raw(x0) is the pixel-by-pixel data value received by the sensor at phase 0°, and Raw(x 90 ) is the per-pixel data value received by the sensor at phase 90°, and Raw(x 180 ) is the per-pixel data value received by the sensor at phase 180°, and Raw(x 270 ) may be the pixel-by-pixel data value received by the sensor at phase 270°.

[0055] The operation unit (13) can store a table. The table may store first correction coefficients corresponding to the object's pose, the distance between the camera and the object, and the object's shape information. The operation unit (13) can correct distance errors within the depth image based on the first correction coefficients selected based on the IR image. The operation unit (13) may include a processing unit and a memory to perform the processes of generating the IR image and the depth image and correcting the depth image.

[0056] FIG. 2 is a drawing showing a camera device according to another embodiment of the present invention.

[0057] Referring to FIG. 2, a camera device (1) according to an embodiment of the present invention may include a ToF camera module (10), an RGB camera, and a processing unit.

[0058] The ToF camera module (10) may have the same configuration as the ToF camera module (10) described above with reference to FIG. 1, excluding the computation unit. Accordingly, a detailed description of the ToF camera module (10) is omitted.

[0059] The RGB camera module (20) may be a camera module that generates a color image of an object. To this end, the RGB camera module (20) may include a lens, a filter, and a sensor. In this case, the sensor may be a CMOS (complementary metal oxide semiconductor) sensor or a CCD (charge coupled device) sensor.

[0060] The RGB camera module (20) can capture an object by interacting with the ToF camera module (10). For example, the RGB camera module (20) can capture an object at the time the ToF camera module (10) captures the object.

[0061] The processing unit (30) can receive an electrical signal from the ToF camera module (10) to generate a two-dimensional IR image and a three-dimensional depth image. The processing unit (30) can receive an electrical signal from the RGB camera module (20) to generate a two-dimensional color image.

[0062] The operation unit (30) can store a table. The table may store a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information. The operation unit (30) can correct the distance error within the depth image based on the first correction coefficient selected based on the IR image. The operation unit (30) can detect color information corresponding to the feature point detected from the IR image from the color image. The operation unit (30) can correct the feature point based on a second correction coefficient corresponding to the color information.

[0063] The computation unit (30) may include a computation processing unit and memory to perform the generation of IR images, depth images, and color images, and the correction process of depth images and feature points. For example, if the camera device (1) is implemented as a terminal device such as a smartphone, the computation unit (30) may be implemented as the application processor (AP) and memory of the terminal device.

[0064] FIG. 3 is a configuration diagram of a depth image correction device according to an embodiment of the present invention.

[0065] The depth image correction device (300) according to an embodiment of the present invention can be implemented through the computation unit described above. For example, the depth image correction device (300) according to an embodiment of the present invention can be implemented through a computation processing unit (processor) and memory.

[0066] Referring to FIG. 3, a depth image correction device (300) according to an embodiment of the present invention may include a detection unit (310), a selection unit (320), a calculation unit (330), a depth correction unit (340), a feature point correction unit (350), and an output unit (360).

[0067] The detection unit (310) can detect feature points corresponding to an object from a first image, which is an IR image of an object.

[0068] The first image may refer to a two-dimensional image generated based on an electrical signal generated by a ToF camera module. According to one embodiment, the depth image correction device (300) may generate the first image by generating an amplitude image or an intensity image based on an electrical signal generated by a ToF camera module.

[0069] A feature point may refer to a point in an image that can serve as a feature or a point containing important information. For example, a feature point may be a corner point or an edge within the image. As another example, a feature point may be a point within the image where the pixel value shows a rapid change. The detection unit (310) may include a feature point detection algorithm for detecting feature points of the first image. According to one embodiment, the feature point detection algorithm may include algorithms for detecting feature points such as the Harris corner detector, SIFT (Scale Invariant Feature Transform), and FAST (Features from Accelerated Segment Test). The feature point detection algorithm may be stored in the memory of the computation unit.

[0070] The selection unit (320) can select a first correction coefficient of shape information corresponding to a feature point from a table in which a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information is stored. The table may be a lookup table and may be stored in the memory of the operation unit.

[0071] The calculation unit (330) can calculate the distance error for an object based on the first correction coefficient. The calculation unit (330) can calculate the distance error for an object using the following mathematical formula 5.

[0072]

[0073] Here, c represents the speed of light, and f represents the frequency of the output light of the ToF camera module, represents the first correction factor.

[0074] The depth correction unit (340) can correct the depth information of the second image by applying a distance error to the second image that measures depth information about the object.

[0075] The second image may refer to a three-dimensional depth image generated based on an electrical signal generated by a ToF camera module. According to one embodiment, the depth image correction device (300) can generate the second image by generating depth information based on an electrical signal generated by a ToF camera module.

[0076] The feature point correction unit (350) can detect color information corresponding to the feature point from the third image, which is an RGB image of an object, and correct the feature point based on the color information.

[0077] The feature point correction unit (350) can select a second correction coefficient corresponding to the detected color information from a table in which a second correction coefficient corresponding to the color reflectance is stored, correct a pixel value corresponding to the feature point based on the selected second correction coefficient, and correct the feature point based on the corrected pixel value for the feature point.

[0078] The output unit (360) can select and provide an object type corresponding to a feature point from a table. Information about the object type can be stored in the table.

[0079] FIG. 4 is a drawing for explaining a table according to an embodiment of the present invention.

[0080] Different types of objects may differ in light reflectivity, shape, and curvature. For example, stainless steel kitchen tools and glass cups may have different light reflectivity, and their shapes and curvatures also differ. Furthermore, even for the same object, light reflectivity, shape, and curvature can vary depending on the object's orientation. For instance, if the object is a cup, its shape and curvature may differ depending on whether the handle is positioned in the direction of the shot or perpendicular to it. These object characteristics can cause different measurement errors in the depth information measured by the ToF camera module and may lead to the loss of information regarding specific parts. Since object characteristics differ, error correction through methods such as hardware calibration is difficult. However, if these errors are not corrected, it becomes impossible to accurately represent the depth information of the objects. To solve these problems, a depth image correction device according to an embodiment of the present invention corrects depth information for an object using a table in which a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information is stored.

[0081] Table 1 below shows an example of a table according to an embodiment of the present invention.

[0082] Object types distance posture Shape information First correction factor cup d1 A f1(x1, y1, z1), f1(x2, y2, z2), … , f1(xn, yn, zn) Δφ1 cup d2 A f2(x1, y1, z1), f2(x2, y2, z2), … , f2(xn, yn, zn) Δφ2 cup d3 A f3(x1, y1, z1), f3(x2, y2, z2), … , f3(xn, yn, zn) Δφ3 cup d1 B f4(x1, y1, z1), f4(x2, y2, z2), … , f4(xn, yn, zn) Δφ4 … … … … … doll d1 C fm(x1, y1, z1), fm(x2, y2, z2), … , fm(xn, yn, zn) Δφm … … … … …

[0083] Referring to Table 1, the table can store a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information.

[0084] Object types indicate the types of objects. Object types may include not only intangible objects such as cups and dolls shown in Table 1, but also specific parts of the human body such as hands and feet.

[0085] The distance represents the distance between the camera and the object as illustrated in FIG. 4. For example, the distance may represent the shortest distance from the sensor surface of the ToF camera to the object. However, this is merely an example, and the distance may also represent the distance between a specific point of the ToF camera and a specific point of the object, and design modifications may be possible by those skilled in the art.

[0086] A pose represents the posture of an object. For example, if the object is a chair, the camera can photograph the chair from various directions, such as the front, back, top, and side, and the captured image varies depending on the direction. In this case, the direction the camera is looking at the object may be the object's pose. The table can store each pose by indexing it as shown in Table 1 of the object, but this is merely an example, and each pose can be stored in various ways.

[0087] Shape information represents the shape information of an object. According to one embodiment, shape information may refer to coordinate values ​​for the boundary (i.e., edge) of an object, as illustrated in FIG. 4. Shape information may be coordinate values ​​for the entire boundary of an object, or coordinate values ​​for key feature points within the boundary of an object. In Table 1, for convenience of explanation, the shape information of an object is represented by 1 to n coordinate values, but the coordinate values ​​representing the shape information of an object may vary depending on the object, and may also vary depending on the orientation or distance of the object. In this case, the coordinate values ​​may be three-dimensional coordinate values. That is, they may include not only coordinate values ​​for the x-axis and y-axis, but also coordinate values ​​for the z-axis, which is a depth value.

[0088] The first correction coefficient may be a distance correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information. According to one embodiment, the first correction coefficient may include a distance correction coefficient for a specific point. For example, the first correction coefficient may include a correction coefficient corresponding to the coordinate values ​​of the object's shape information. As another example, the first correction coefficient may include a correction coefficient for the entire shape of the object. That is, the first correction coefficient may include a correction coefficient corresponding to the pixels of the entire object shape. The correction coefficient may be a numerical value, but is not limited thereto. The correction coefficient may also be represented by a mathematical algorithm. That is, the distance correction coefficient for the entire object shape may be represented as a mathematical function.

[0089] Meanwhile, the table may be generated by machine learning. The table may be generated through a machine learning algorithm that generates a first correction coefficient using the object's edge information and depth information detected from at least one of the IR image or RGB image of the object captured at a previous time point as training data. For example, the machine learning algorithm may be a supervised learning-based machine learning algorithm. As another example, the machine learning algorithm may be an unsupervised learning-based machine learning algorithm. The machine learning algorithm may be stored in a computation unit. Training data may be generated using IR images or RGB images captured at multiple poses and multiple distances for each of multiple types of objects. That is, training data may be generated using at least one of IR images or RGB images captured at various poses and various distances for various types of objects. Meanwhile, the table may generate training data using the first, second, and third images captured at the current time point as well as the previous time point. The generated training data is input into the machine learning algorithm, and the machine learning algorithm may update the table based on the training data generated at the current time point.

[0090] FIG. 5 is a flowchart of a depth image correction method according to a first embodiment of the present invention.

[0091] First, the detection unit can detect feature points corresponding to an object from a first image, which is an IR image of the object (S510). At this time, the feature points may be coordinate values ​​of the boundary points of the object captured in the first image. The feature points may be coordinate values ​​for specific points among the boundary points of the object, or coordinate values ​​for the whole. This may vary depending on the algorithm for detecting feature points, and design changes are possible by those skilled in the art.

[0092] Then, the selection unit can select a first correction coefficient of shape information corresponding to a feature point from a table in which first correction coefficients corresponding to the object's pose, the distance between the camera and the object, and the object's shape information are stored (S520). According to one embodiment, the selection unit can select a first correction coefficient based on similarity by comparing the coordinate value of the feature point with the coordinate value included in the object's shape information stored in the table. For example, the selection unit can select a first correction coefficient corresponding to the shape information with the highest similarity calculated within a group in which the similarity between the coordinate value of the feature point and the coordinate value included in the object's shape information is greater than a predetermined value.

[0093] The calculation unit can calculate a distance error for an object based on a first correction coefficient (S530). According to one embodiment, the calculation unit can calculate a distance error for all pixels representing an object based on a first correction coefficient.

[0094] The depth correction unit can correct the depth information of the second image by applying a distance error to the second image in which depth information about the object has been measured (S540). For example, the depth correction unit can correct the depth information of the second image by adding (or subtracting) the distance error from the depth information of the pixel corresponding to the object in the second image.

[0095] The output unit can output a second image with depth information corrected. For example, the output unit can transmit the second image with depth information corrected to the display unit of a user terminal to provide it to the user. At this time, the output unit can transmit the corrected second image including information about the object. If the object in the second image is a cup, the output unit can output the object in the second image including the object type (category) of "cup." If there are multiple object types, the output unit can provide the object types for the multiple objects together when outputting the second image. At this time, the object types can be obtained from a table. The object types can be provided to the output unit by selecting an object type corresponding to a feature point from the table.

[0096] FIG. 6 is a flowchart of a depth image correction method according to a second embodiment of the present invention.

[0097] Referring to FIG. 6, the depth image correction method according to the second embodiment of the present invention may be a depth image correction method that further includes a plurality of steps for correcting feature points in the depth image correction method according to the first embodiment described with reference to FIG. 5.

[0098] First, the detection unit can detect feature points corresponding to an object from a first image, which is an IR image of the object (S610). At this time, the feature points may be coordinate values ​​of the boundary points of the object captured in the first image. The feature points may be coordinate values ​​for specific points among the boundary points of the object, or coordinate values ​​for the whole. This may vary depending on the algorithm for detecting feature points, and design changes are possible by those skilled in the art.

[0099] Then, the feature point correction unit can detect color information corresponding to the feature point from the third image, which is an RGB image of the object (S620). That is, the feature point correction unit can detect color information of the pixel corresponding to the feature point in the third image. Here, color information may mean color information having a wavelength in the visible bandwidth.

[0100] The feature point correction unit can select a second correction coefficient corresponding to the detected color information from a table in which a second correction coefficient corresponding to the color reflectance is stored (S630). The table may store not only the first correction coefficient but also a second correction coefficient corresponding to the color information. An object may have various colors, and each color has a light reflectance in each wavelength band. For example, when the light-emitting part of a ToF camera module irradiates light in the 850 to 1100 nm band onto an object, the amount of light reflected from the blue surface of the object and the amount of light reflected from the black surface of the object will differ even at the same distance. If this phenomenon is not taken into account, a problem may arise where an object is detected as a feature point due to the light reflectance even though it is not a feature point. To solve this problem, the table according to an embodiment of the present invention may store a second correction coefficient based on the light reflectance for each color.

[0101] The feature point correction unit can correct the pixel value corresponding to the feature point based on the selected second correction coefficient (S640). The feature point correction unit can calculate the pixel error value of each feature point using the second correction coefficient and correct the pixel value of the feature point by adding (or subtracting) the pixel error value to the pixel value of the feature point.

[0102] The feature point correction unit can correct the feature point based on the corrected pixel value of the feature point (S650). According to one embodiment, the feature point correction unit can remove the feature point if the corrected pixel value of the feature point is lower than a preset threshold. For example, if the corrected pixel value of 3 feature points out of 10 feature points is lower than a preset threshold, the feature point correction unit can determine that the 3 feature points are not the ones with pixel values ​​lower than the threshold and remove them.

[0103] Then, the selection unit can select a first correction coefficient of shape information corresponding to a feature point from a table in which first correction coefficients corresponding to the object's pose, the distance between the camera and the object, and the object's shape information are stored (S660). Here, the feature point may be the feature point corrected above. According to one embodiment, the selection unit can select a first correction coefficient based on similarity by comparing the coordinate value of the feature point with the coordinate value included in the object's shape information stored in the table. For example, the selection unit can select a first correction coefficient corresponding to the shape information with the highest similarity calculated within a group in which the similarity between the coordinate value of the feature point and the coordinate value included in the object's shape information is greater than a predetermined value.

[0104] The calculation unit can calculate a distance error for an object based on a first correction coefficient (S670). According to one embodiment, the calculation unit can calculate a distance error for all pixels representing an object based on a first correction coefficient.

[0105] The depth correction unit can correct the depth information of the second image by applying a distance error to the second image in which depth information about the object has been measured (S680). For example, the depth correction unit can correct the depth information of the second image by adding (or subtracting) the distance error from the depth information of the pixel corresponding to the object in the second image.

[0106] The output unit can output a second image with depth information corrected. For example, the output unit can transmit the second image with depth information corrected to the display unit of a user terminal to provide it to the user. At this time, the output unit can transmit the corrected second image including information about the object. If the object in the second image is a cup, the output unit can output the object in the second image including the object type (category) of "cup." If there are multiple object types, the output unit can provide the object types for the multiple objects together when outputting the second image. At this time, the object types can be obtained from a table. The object types can be provided to the output unit by selecting an object type corresponding to a feature point from the table.

[0107] Figure 7 is a simulation result according to an embodiment of the present invention.

[0108] FIG. 7(a) is a drawing showing a second image without distance error correction, and FIG. 7(b) is a drawing showing a second image with distance error correction.

[0109] The object within the dotted lines in FIG. 7(a) and FIG. 7(b) is a curtain, which is made of a material with high light reflectivity and has many curves formed by wrinkles for its intended use. Accordingly, when the distance error is not corrected as shown in FIG. 7(a), it can be seen that the distance information between adjacent pixels appears large despite the same distance, and in some areas, the distance information is lost. However, when the distance error is corrected by the depth image correction device according to an embodiment of the present invention as shown in FIG. 7(b), distance information that accurately reflects the shape appears (distance information between adjacent pixels is uniform), and it can be seen that there are no areas where the distance information is lost.

[0110] Although the invention has been described above with reference to embodiments, this is merely illustrative and does not limit the invention. Those skilled in the art will understand that various modifications and applications not exemplified above are possible within the scope of the essential characteristics of the embodiments. For example, each component specifically shown in the embodiments may be modified and implemented. Furthermore, differences related to such modifications and applications should be interpreted as being included within the scope of the invention as defined in the appended claims. Explanation of the symbols

[0111] 300: Depth image correction device 310 : Detector 320 : Selection 330 : Output section 340 : Depth correction unit 350 : Feature point correction unit 360 : Output section

Claims

Claim 1 A depth image correction device comprising: a detection unit for detecting feature points corresponding to an object from a first image, which is an IR image of an object; a selection unit for selecting a first correction coefficient of shape information corresponding to the feature points from a table storing a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information; a calculation unit for calculating a distance error for the object based on the first correction coefficient; and a depth correction unit for correcting the depth information of the second image by applying the distance error to a second image in which depth information for the object is measured; wherein the first image and the second image are depth information for the object measured by different methods. Claim 2 In claim 1, the depth image correction device is generated through a machine learning algorithm that generates the first correction coefficient using the boundary information of the object detected from at least one of an IR image or an RGB image taken of the object at a previous time point and the depth information of the object as learning data. Claim 3 In paragraph 2, the above-mentioned learning data is a depth image correction device generated using IR images or RGB images captured at multiple poses and multiple distances for each of multiple types of objects. Claim 4 A depth image correction device comprising: a feature point correction unit that detects color information corresponding to the feature point from a third image, which is an RGB image of the object captured in claim 1, and corrects the feature point based on the color information. Claim 5 In claim 4, the feature point correction unit selects a second correction coefficient corresponding to the color information detected from the table in which a second correction coefficient corresponding to the color reflectance is stored, corrects a pixel value corresponding to the feature point based on the selected second correction coefficient, and corrects the feature point based on the corrected pixel value of the feature point. Claim 6 A depth image correction device comprising: an output unit that selects and provides an object type corresponding to the feature point from the table in claim 1. Claim 7 A depth image correction method comprising: a step of detecting a feature point corresponding to an object from a first image, which is an IR image of an object; a step of selecting a first correction coefficient of shape information corresponding to the feature point from a table storing a first correction coefficient corresponding to the object's pose, the distance between the camera and the object, and the object's shape information; a step of calculating a distance error for the object based on the first correction coefficient; and a step of applying the distance error to a second image in which depth information for the object is measured to correct the depth information of the second image; wherein the first image and the second image are images in which depth information for the object is measured by a different method. Claim 8 In claim 7, the depth image correction method is generated through a machine learning algorithm that generates the first correction coefficient using the boundary information of the object detected from at least one of the IR image or RGB image of the object captured at a previous time point and the depth information of the object as training data. Claim 9 In claim 8, the above-mentioned learning data is a depth image correction method generated using IR images or RGB images taken at multiple poses and multiple distances for each of multiple types of objects. Claim 10 A depth image correction method according to claim 7, comprising: a step of detecting color information corresponding to the feature point from a third image, which is an RGB image of the object; and a step of correcting the feature point based on the color information. Claim 11 A depth image correction method according to claim 10, wherein the step of correcting the feature point comprises: a step of selecting a second correction coefficient corresponding to the color information detected from a table in which a second correction coefficient corresponding to the color reflectance is stored; a step of correcting a pixel value corresponding to the feature point based on the selected second correction coefficient; and a step of correcting the feature point based on the corrected pixel value for the feature point. Claim 12 A depth image correction method comprising the step of providing a selection of an object type corresponding to the feature point from a previously stored table in claim 7.

Citation Information

Patent Citations

  • Method and apparatus for recovering depth value of depth image

    US20130176391A1

  • Information processing apparatus and information processing method

    US20180075315A1

  • Method for Processing a Raw Image of a Time-of-Flight Camera, Image Processing Apparatus and Computer Program

    US20190147624A1

  • Depth acquisition device, depth-acquiring method and program

    WO2020059448A1

  • Depth from time-of-flight using machine learning

    US20180129973A1