Method and apparatus for processing image

Abstract

One embodiment of the present invention provides a method for processing an image, the method comprising the steps of: acquiring an original depth image captured by a ToF sensor; acquiring an RGB image captured by an RGB camera; setting a low reflection region in the original depth image; acquiring luminance data of a matching region corresponding to the low reflection region in the RGB image; and predicting image information in the low reflection region of the original depth image by using the acquired luminance data.

Description

Image processing method and device

[0001] Embodiments of the present invention relate to an image processing method and apparatus, and more particularly, to an image processing method and apparatus capable of improving the quality of a depth image by predicting and restoring image information of a low-reflection area in a depth image.

[0002] Unlike conventional two-dimensional images, 3D images simultaneously provide information about depth and spatial shape, similar to what humans see in real life. These 3D images not only enhance the quality of visual information but also convey a variety of information with a realistic quality, making them useful in numerous fields, including autonomous driving, gaming, education, medicine, and information and communications.

[0003] In particular, with the development and increasing demand for 3D display devices, the demand for high-resolution, natural-looking 3D images is increasing, and research is underway to develop methods for generating more effective 3D images. Among these 3D imaging technologies, technologies based on depth imaging have recently been developed and applied in various fields, and various attempts are being made to improve the quality of depth images.

[0004] Depth images can be acquired using a ToF (Time of Flight) sensor. A ToF sensor uses the time of flight—the time it takes for light directed toward a subject to be reflected and returned—to measure the distance between the sensor and the subject and generate a depth image.

[0005] However, conventional ToF sensors have a problem in that their recognition rate is low for low-reflective subjects that reflect little light. Low-reflective subjects absorb the light irradiated toward them and reflect only a very small amount of light. In the case of such low-reflective subjects, the amount of light reflected and returned by the subject is insufficient, making it difficult for the ToF sensor to accurately recognize depth information. Therefore, there was a problem in that it was difficult to obtain accurate image information using only the depth image acquired through the ToF sensor.

[0006] Embodiments of the present invention are intended to solve the above-mentioned problems, and an object thereof is to provide an image processing method and device capable of predicting and restoring image information for a low-reflection area included in a depth image.

[0007] However, these tasks are exemplary and the scope of the present invention is not limited thereby.

[0008] One embodiment of the present invention provides an image processing method including the steps of: acquiring an original depth image captured by a ToF sensor; acquiring an RGB image captured by an RGB camera; setting a low-reflection area in the original depth image; acquiring luminance data of a matching area corresponding to the low-reflection area in the RGB image; and predicting image information in the low-reflection area of ​​the original depth image using the acquired luminance data.

[0009] The image processing method and device according to embodiments of the present invention can effectively predict and restore image information for a low-reflection area included in a depth image using luminance data of an RGB image captured by an RGB camera.

[0010] In addition, the image processing method and device according to embodiments of the present invention can obtain a more accurate depth image by restoring image information of a low-reflection area in a depth image using luminance data of an RGB image.

[0011] Of course, the scope of the present invention is not limited by these effects.

[0012] FIG. 1 is a diagram schematically illustrating the operation of an image processing device according to one embodiment of the present invention.

[0013] Figure 2 is a diagram illustrating a method for setting a low-reflection area in an original depth image and setting a matching area corresponding to the low-reflection area in an RGB image.

[0014] Figure 3 is a diagram illustrating a method for dividing a low-reflection area in an original depth image into two or more luminance-similar areas.

[0015] Figure 4 is a diagram illustrating a method for obtaining multiple frame data from an original depth image acquired in a time-series manner.

[0016] Figure 5 is a diagram illustrating a method for setting a surrounding area adjacent to a low-reflection area in an original depth image.

[0017] Figure 6 is a diagram illustrating a method for obtaining luminance directionality of a matching area in an RGB image and predicting image information of a low-reflection area in an original depth image using the luminance directionality.

[0018] FIG. 7 is a flowchart illustrating an image processing method according to one embodiment of the present invention.

[0019] Figure 8 is a block diagram showing the configuration of an electronic device according to one embodiment of the present invention.

[0020] One embodiment of the present invention provides an image processing method including the steps of: acquiring an original depth image captured by a ToF sensor; acquiring an RGB image captured by an RGB camera; setting a low-reflection area in the original depth image; acquiring luminance data of a matching area corresponding to the low-reflection area in the RGB image; and predicting image information in the low-reflection area of ​​the original depth image using the acquired luminance data.

[0021] In one embodiment of the present invention, the low-reflection area can be determined according to the amount of light received by the ToF sensor.

[0022] In one embodiment of the present invention, the step of setting the low-reflection area may set the low-reflection area after removing the background area from the original depth image.

[0023] In one embodiment of the present invention, the step of predicting the image information may include the step of dividing the low-reflection area into two or more luminance-similar areas using luminance data of the matching area, the step of obtaining a plurality of frame data from the original depth image acquired in time series for each of the divided luminance-similar areas, and the step of predicting the image information of each of the luminance-similar areas using the plurality of frame data.

[0024] In one embodiment of the present invention, the step of predicting the image information of each luminance-similar region may include the step of extracting valid data from the frame data, the step of calculating an average value of the valid data for each luminance-similar region, and the step of determining a predicted value of the image information of each luminance-similar region based on the calculated average value.

[0025] In one embodiment of the present invention, the step of predicting the image information may include the step of obtaining depth information of a peripheral area adjacent to the low-reflection area from the original depth image, the step of obtaining luminance directionality of the low-reflection area using luminance data of the matching area, and the step of predicting image information of the low-reflection area using the depth information of the peripheral area and the luminance directionality of the low-reflection area.

[0026] Another embodiment of the present invention provides a computer program stored on a computer-readable recording medium for executing any one of the methods according to the embodiments of the present invention in combination with hardware.

[0027] Another embodiment of the present invention provides an image processing device comprising a memory storing at least one program and at least one processor that performs image processing by executing the at least one program, wherein the memory includes instructions causing the at least one processor to execute a step of acquiring an original depth image captured by a ToF sensor, a step of acquiring an RGB image captured by an RGB camera, a step of setting a low-reflection area in the original depth image, a step of acquiring luminance data of a matching area corresponding to the low-reflection area among the RGB images, and a step of predicting image information in the low-reflection area of ​​the original depth image using the acquired luminance data.

[0028] Another embodiment of the present invention provides an electronic device including a ToF sensor that generates an original depth image, an RGB camera that generates an RGB image, and an image processing device that sets a low-reflection area in the original depth image, obtains luminance data of a matching area corresponding to the low-reflection area in the RGB image, and predicts image information in the low-reflection area of ​​the original depth image using the obtained luminance data.

[0029] Hereinafter, the following embodiments will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same drawing reference numerals, and redundant descriptions thereof will be omitted.

[0030] These embodiments are capable of various modifications. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of these embodiments, as well as the methods for achieving them, will become clearer with reference to the detailed descriptions below, along with the drawings. However, these embodiments are not limited to the embodiments disclosed below and may be implemented in various forms.

[0031] In the drawings, parts unrelated to the description are omitted to clearly explain the present invention, and similar parts are designated by similar drawing reference numerals throughout the specification.

[0032] In the examples below, the terms first, second, etc. are not used in a limiting sense, but are used for the purpose of distinguishing one component from another.

[0033] In the examples below, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0034] In the examples below, terms such as “include” or “have” mean that a feature or component described in the specification exists, and do not exclude in advance the possibility that one or more other features or components may be added.

[0035] In the examples below, when a part such as a unit, region, or component is said to be on or above another part, this includes not only the case where it is directly above the other part, but also the case where another unit, region, component, etc. is interposed in between.

[0036] In the examples below, terms such as connect or combine do not necessarily mean a direct and / or fixed connection or combination of two members, unless the context clearly indicates otherwise, and do not exclude the presence of another member between the two members.

[0037] For convenience of explanation, the sizes of components in the drawings may be exaggerated or reduced. For example, the sizes and thicknesses of each component shown in the drawings are arbitrarily indicated for convenience of explanation, and thus the following embodiments are not necessarily limited to those shown.

[0038] FIG. 1 is a diagram schematically showing the operation of an image processing device (100) according to one embodiment of the present invention.

[0039] Referring to FIG. 1, the image processing device (100) can generate a restored depth image (103) based on an original depth image (101) and an RGB image (102).

[0040] The original depth image (101) can be captured by a ToF (Time of Flight) sensor. The ToF sensor can include a light source that irradiates light to a subject and can generate an original depth image (101) by receiving light reflected from the subject. In addition, the ToF sensor can include a processor that controls the ToF sensor, calculates the phase of light reflected and measured from the subject, calculates depth information of the subject, and generates a depth image, and can include a memory that stores the results processed by the processor.

[0041] The light source can be implemented as a light emitting diode (LED) or a laser diode (LD), and can irradiate light in the infrared (IR) or near infrared (near IR) band to the subject. The light source can sequentially irradiate light of different phases to the subject.

[0042] The light of each phase is reflected by the surface of the subject. The reflected light of different phases is incident on the ToF sensor. The processor uses the light signals of different phases detected by the ToF sensor to generate a depth image.

[0043] More specifically, the time-of-flight of light emitted from a light source and reflected by the subject can be determined based on the distance between the light source and the subject. By calculating the time-of-flight corresponding to the difference between the time the light is emitted from the light source and the time the light is sensed by the ToF sensor, the distance between the subject and the ToF sensor can be calculated.

[0044] The processor generates infrared images corresponding to each phase based on the intensity of the sensed infrared light, calculates the time of flight using the infrared images of each phase, and generates a depth image with depth values ​​based on the calculated time of flight. The ToF sensor can generate a depth image of a scene including the subject and its surroundings by calculating the depths of the subject and its surroundings using this ToF principle.

[0045] The original depth image (101) captured by the ToF sensor may include image information such as depth information of the subject and frame data.

[0046] The RGB image (102) can be captured by an RGB camera. Here, the RGB camera is not limited to a specific type and may refer to a capturing device having an RGB sensor. In addition, to ensure correspondence between the RGB image (102) and the original depth image (101), the ToF sensor and the RGB camera can capture images at the same location. In this case, the ToF sensor and the RGB camera may be provided as separate devices, but are not limited thereto, and may be implemented as a single device.

[0047] An RGB image (102) captured by an RGB camera may include image information such as brightness data.

[0048] The image processing device (100) can predict and restore image information in a low-reflection area included in the original depth image (101) based on the original depth image (101) and the RGB image (102), and can generate a restored depth image (103). Here, the low-reflection area refers to a portion of the original depth image (101) captured by the ToF sensor where the amount of light received by the ToF sensor is small and sufficient image information is not acquired.

[0049] For example, if the color of the subject is black, the subject absorbs most of the light, and the amount of light reflected from the subject and returned to the ToF sensor is very small, and the ToF sensor cannot obtain sufficient image information about the subject. Therefore, the low-reflection area corresponding to the subject in the original depth image (101) contains almost no image information and may appear as darkness in the original depth image (101).

[0050] An image processing device (100) can predict and restore image information in a low-reflection area included in an original depth image (101) using luminance data of an RGB image (102), and can generate a restored depth image (103) with improved image information in the low-reflection area.

[0051] Hereinafter, an example in which an image processing device (100) generates a restored depth image (103) based on an original depth image (101) and an RGB image (102) will be described with reference to FIGS. 2 to 4.

[0052] FIG. 2 is a drawing showing a method of setting a low-reflection area (T) in an original depth image (102) and setting a matching area (M) corresponding to the low-reflection area (T) in an RGB image (101).

[0053] Referring to FIG. 2, a low-reflection area (T) can be set in the original depth image (102) acquired from the ToF sensor. The low-reflection area (T) can be determined by the amount of light received by the ToF sensor. For example, the low-reflection area (T) can be set as a portion where the amount of light received by the ToF sensor is less than a preset value.

[0054] Meanwhile, as an embodiment, before setting the low-reflection area (T) in the original depth image (102), a process of removing image information of the background area may be performed first. Here, the background area is an area corresponding to the background around the subject in the original depth image (102) rather than the subject itself, and may correspond to the part furthest from the ToF sensor, but is not limited thereto. The background area may be determined using depth information of the original depth image (102), and may also be determined using brightness information of the RGB image (101), etc., without being limited thereto.

[0055] For example, after determining the I region and the II region corresponding to the bottom portion in the original depth image (102) of FIG. 2 as the background region, the image information of the I region and the II region can be removed from the original depth image (102). The original depth image (102) from which the image information of the background region has been removed can set the low-reflection region (T) more accurately.

[0056] Next, a matching area (M) corresponding to the low-reflection area (T) of the original depth image (102) can be set in the RGB image (101) acquired from the RGB camera. Since the RGB image (101) includes luminance data, luminance data of the matching area (M) can be acquired from the RGB image (101).

[0057] Figure 3 is a drawing showing a method for dividing a low-reflection area (T) in an original depth image (102) into two or more luminance-similar areas.

[0058] Referring to FIG. 3, a low-reflection area (T) set in an original depth image (102) can be divided into a plurality of luminance-similar areas (A, B, C, D). An RGB image (101) includes luminance data, and based on the luminance data, a matching area (M) of the RGB image (101) can be divided into a preset number of luminance-similar areas (A, B, C, D) determined by a similar range of luminance values. At this time, the number of luminance-similar areas may be selected as an appropriate number by calculation, but is not limited thereto, and may be arbitrarily selected by a user.

[0059] Next, by matching the RGB image (101) and the original depth image (102), the low-reflection area (T) of the original depth image (102) can be divided into a plurality of luminance-similar areas (A, B, C, D) so as to correspond to the luminance-similar areas (A, B, C, D) of the divided matching area (M).

[0060] FIG. 4 is a diagram illustrating a method for obtaining multiple frame data from an original depth image (102) acquired in a time series manner.

[0061] Referring to FIG. 4, a plurality of frame data (401) can be acquired for each luminance-like region from the original depth image (102). The frame data (401) corresponds to each frame in which the luminance-like region is captured in the original depth image (102) acquired in a time series manner, and each frame data (401) of the luminance-like region contains very little image information.

[0062] A predetermined number (N) of such frame data (401) can be collected, and the N collected frame data (401) can be overlapped to generate accumulated data (402) for a luminance-like region. The accumulated data (402) can include all image information included in each of the N frame data (401). The accumulated data (402) is acquired for each luminance-like region.

[0063] Meanwhile, in one embodiment, the number (N) of frame data (401) collected can be determined according to the movement data of the object. That is, the movement of the object corresponding to the shooting target in the luminance-similar area can be detected, movement data for the object can be generated, and the number (N) of frame data (401) collected can be adjusted based on the movement data. For example, a greater number of frame data (401) can be collected for a moving object than for a stationary object, and a greater number of frame data (401) can be collected for a fast-moving object than for a slow-moving object.

[0064] In addition, to improve prediction accuracy, valid data is filtered from multiple frame data (401). In one embodiment, the valid data may be obtained by deriving maximum and minimum values ​​from depth information in the remaining areas of the original depth image (102) excluding the low-reflection area and the background area, and extracting only frame data (401) located between the maximum and minimum values ​​from the multiple frame data (401).

[0065] Next, image information is predicted for each luminance-like region using the extracted valid data. In one embodiment, the predicted image information may be depth information predicted based on the average value of the depth information of the valid data for each luminance-like region and the average value calculated. Using the predicted image information, each luminance-like region can be restored and a restored depth image can be generated.

[0066] Hereinafter, another example in which an image processing device (100) generates a restored depth image (103) based on an original depth image (101) and an RGB image (102) will be described with reference to FIGS. 5 and 6.

[0067] FIG. 5 is a drawing showing a method for setting a surrounding area (S1, S2) adjacent to a low-reflection area (T) in an original depth image (101), and FIG. 6 is a drawing showing a method for obtaining the luminance directionality of a matching area (M) in an RGB image (102) and predicting image information of a low-reflection area (T) in an original depth image using the luminance directionality.

[0068] Referring to FIGS. 5 and 6, when the low-reflection subject in the original depth image (101) has a shape with symmetry such as a cylinder, a restored depth image (103) can be generated using image information of an area adjacent to the low-reflection subject and luminance data of an RGB image.

[0069] To give a specific example, if the original depth image (101) includes a low-reflection area (T) of a cylinder connected to a ceiling surface (U) and a floor surface (G), a first peripheral area (S1) that appears adjacent to the low-reflection area (T) on the ceiling surface (U) and a second peripheral area (S2) that appears adjacent to the low-reflection area (T) on the floor surface (G) can be set using the original depth image (101). The peripheral areas can be set along the edge of the low-reflection subject. Next, depth information of the set peripheral areas (S1, S2) is acquired.

[0070] In addition, the luminance data of the matching area (M) corresponding to the low-reflection area (T) can be obtained using the RGB image (102), and the luminance directionality of the low-reflection area (T) can be obtained using the luminance data of the matching area (M). Here, the luminance directionality may mean the direction in which a part having the same luminance value extends. In the case of a low-reflection subject having symmetry such as a cylinder, the luminance data of the RGB image (102) also has directionality, and the luminance directionality can be calculated.

[0071] Therefore, after calculating the luminance directionality of the matching area (M), this is determined as the luminance directionality of the low-reflection area (T), and the image information of the low-reflection area (T) can be predicted using the luminance directionality of the low-reflection area (T). At this time, the image information of the low-reflection area can be linearly predicted between the set first peripheral area (S1) and the second peripheral area (S2) using the depth information of the first peripheral area (S1) and the second peripheral area (S2) and the luminance directionality of the low-reflection area (T). The image information of the low-reflection area predicted in this way can be used to restore the low-reflection area (T), and a restored depth image can be obtained by restoring the low-reflection area (T).

[0072] FIG. 7 is a flowchart illustrating an image processing method according to one embodiment of the present invention.

[0073] Referring to FIG. 7, an image processing method according to an embodiment of the present invention may include a step of obtaining an original depth image captured by a ToF sensor (S100), a step of obtaining an RGB image captured by an RGB camera (S200), a step of setting a low-reflection area in the original depth image (S300), a step of obtaining luminance data of a matching area corresponding to the low-reflection area in the RGB image (S400), and a step of predicting image information in the low-reflection area of ​​the original depth image using the obtained luminance data (S500).

[0074] In step (S100), the image processing device can acquire an original depth image captured by the ToF sensor. The original depth image captured and acquired by the ToF sensor can include image information such as depth information of the subject and frame data.

[0075] In step (S200), the image processing device can acquire an RGB image captured by an RGB camera. The ToF sensor and the RGB camera can capture images from the same location, and the RGB image and the original depth image can correspond to each other. The RGB image captured by the RGB camera can include image information such as luminance data.

[0076] In step (S300), the image processing device can set a low-reflection area in the original depth image. The low-reflection area can be determined by the amount of light received by the ToF sensor. In one embodiment, the low-reflection area can be set as a portion where the amount of light received by the ToF sensor is less than a preset value.

[0077] Meanwhile, the image processing device may remove the background region from the original depth image prior to setting the low-reflection region. The background region may be determined using, but is not limited to, depth information of the original depth image or brightness information of an RGB image.

[0078] In step (S400), the image processing device can obtain luminance data of a matching area corresponding to a low-reflection area in an RGB image. Specifically, the image processing device can set a matching area in the RGB image corresponding to a low-reflection area set in the original depth image, and obtain luminance data of the matching area.

[0079] In step (S500), the image processing device can predict image information in a low-reflection area of ​​the original depth image using the acquired luminance data.

[0080] In one embodiment, the step (S500) of predicting image information may include a step of dividing a low-reflection area into two or more luminance-similar areas using luminance data of a matching area, a step of acquiring multiple frame data from an original depth image acquired in time series for each of the divided luminance-similar areas, and a step of predicting image information of each luminance-similar area using the acquired multiple frame data.

[0081] Luminance-like regions may be distinguished based on the similarity range of luminance values ​​of luminance data, and the number of luminance-like regions distinguished may be selected as an appropriate number through calculation, but is not limited thereto. In addition, the number of acquired frame data may be determined based on the movement data of the object, but is not limited thereto.

[0082] Additionally, the step of predicting image information of each luminance-similar region may include a step of extracting valid data from frame data, a step of calculating an average value of the valid data for each luminance-similar region, and a step of determining a predicted value of image information of each luminance-similar region based on the calculated average value.

[0083] Specifically, the valid data may be obtained by deriving maximum and minimum values ​​from depth information in an area excluding a low-reflection area and a background area from the original depth image, and extracting only frame data located between the maximum and minimum values ​​from among a plurality of frame data.

[0084] The image processing device can restore each luminance-like region using the image information prediction value of each determined luminance-like region, and can generate a restored depth image.

[0085] In another embodiment, the step (S500) of predicting image information may include a step of obtaining depth information of a peripheral area adjacent to a low-reflection area in an original depth image, a step of obtaining luminance directionality of the low-reflection area using luminance data of a matching area, and a step of predicting image information of the low-reflection area using the depth information of the peripheral area and the luminance directionality of the low-reflection area. The peripheral area refers to a portion adjacent to the low-reflection area in the original depth image, and may be set along the edge of the low-reflection area.

[0086] The image processing device can predict image information of a low-reflection area based on depth information of a surrounding area set in the original depth image and luminance directionality of the low-reflection area obtained from an RGB image, when the original depth image includes a low-reflection area for a subject having symmetry. Here, luminance directionality may mean a direction in which a portion having the same luminance value extends, and the luminance directionality of the low-reflection area can be determined through the luminance directionality of a matching area.

[0087] An image processing device can predict image information of a low-reflection area by using depth information of a surrounding area and luminance directionality of the low-reflection area, and can calculate a predicted value of image information of the low-reflection area by using linearity of luminance directionality.

[0088] The image processing device can restore the low-reflection area from the original depth image using the image information prediction value of the generated low-reflection area, and can generate a restored depth image.

[0089] Figure 8 is a block diagram showing the configuration of an electronic device (10) according to one embodiment of the present invention.

[0090] Referring to FIG. 8, the electronic device (10) may include a ToF sensor (11), an RGB camera (12), and an image processing device (100). The electronic device (10) may be implemented as at least a part of a mobile device such as a mobile phone, a smart phone, a PDA, a netbook, a tablet computer, a laptop computer, etc., a wearable device such as a smart watch, a smart band, a smart glasses, etc., a computing device such as a desktop, a server, etc., a home appliance such as a television, a smart television, a refrigerator, etc., a security device such as a door lock, etc., and a vehicle such as an autonomous vehicle, a smart vehicle, etc.

[0091] The ToF sensor (11) can generate an original depth image using the sensed light of different phases. The original depth image acquired from the ToF sensor (11) may include image information such as depth information and frame data. The RGB camera (12) can generate an RGB image corresponding to the original depth image. The original depth image and the RGB image may be captured at the same location.

[0092] The image processing device (100) may include a processor (110) and a memory (120). The memory (120) is connected to the processor (110) and may store instructions executable by the processor (110), data to be calculated by the processor (110), or data processed by the processor (110). The memory (120) may include a non-transitory computer-readable medium, such as a high-speed random access memory and / or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0093] The processor (110) can execute functions and commands to be executed within the electronic device (10). For example, the processor (110) can process commands stored in the memory (120).

[0094] The processor (110) can execute commands for performing the operations of FIGS. 1 to 7. For example, the processor (110) can set a low-reflection area in the original depth image, acquire luminance data of a matching area corresponding to the low-reflection area in an RGB image, and predict image information in the low-reflection area of ​​the original depth image using the luminance data. In addition, the descriptions of FIGS. 1 to 7 can be applied to the processor (110).

[0095] The embodiments described above can be implemented by hardware components, software components, and / or a combination of hardware components and software components. The method according to the embodiment can be implemented in the form of program commands that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium can store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0096] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will appreciate that various modifications and variations of the embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

[0097] Embodiments according to the present invention can be used to process depth image data including distance information.

Claims

1. A step of acquiring an original depth image captured by a ToF sensor; A step of acquiring an RGB image captured by an RGB camera; A step of setting a low-reflection area in the above original depth image; A step of obtaining luminance data of a matching area corresponding to the low-reflection area among the RGB images; and An image processing method, comprising: a step of predicting image information in a low-reflection area of ​​the original depth image using the acquired luminance data.

2. In paragraph 1, An image processing method, wherein the above-mentioned low-reflection area is determined according to the amount of light received by the ToF sensor.

3. In paragraph 1, An image processing method wherein the step of setting the above low-reflection area comprises setting the above low-reflection area after removing a background area from the original depth image.

4. In paragraph 1, The step of predicting the above image information is: A step of dividing the low-reflection area into two or more luminance-similar areas using luminance data of the matching area; A step of obtaining multiple frame data from the original depth image acquired in a time series manner, for each of the separated luminance-similar regions; and An image processing method, comprising: a step of predicting image information of each of the luminance-similar regions using the plurality of frame data.

5. In paragraph 4, The step of predicting the image information of each of the above luminance-similar regions is: A step of extracting valid data from the above frame data; A step of calculating an average value of the valid data for each luminance similarity area; and An image processing method, comprising: a step of determining a predicted value of the image information of each of the luminance-similar regions based on the calculated average value.

6. In paragraph 1, The step of predicting the above image information is: A step of acquiring depth information of a surrounding area adjacent to the low-reflection area from the original depth image; A step of obtaining the brightness directionality of the low-reflection area by using the brightness data of the matching area; and An image processing method, comprising: a step of predicting image information of the low-reflection area by using depth information of the surrounding area and brightness directionality of the low-reflection area.

7. A computer program stored on a computer-readable recording medium for executing the method of any one of claims 1 to 6 by being combined with hardware.

8. Memory for storing at least one program; and At least one processor for performing image processing by executing at least one program; The above memory is configured such that at least one processor, A step of acquiring an original depth image captured by a ToF sensor; A step of acquiring an RGB image captured by an RGB camera; A step of setting a low-reflection area in the above original depth image; A step of obtaining luminance data of a matching area corresponding to the low-reflection area among the RGB images; and An image processing device, characterized by including commands for executing a step of predicting image information in a low-reflection area of ​​the original depth image using the acquired luminance data.

9. ToF sensor that generates original depth images; RGB camera that produces RGB images; and An electronic device, comprising: an image processing device for setting a low-reflection area in the original depth image, obtaining luminance data of a matching area corresponding to the low-reflection area among the RGB images, and predicting image information in the low-reflection area of ​​the original depth image using the obtained luminance data.

Images