Depth image sensing device, image signal processor, and image signal processing method

Abstract

To provide a depth image sensing device, an image signal processor, and an image signal processing method capable of processing information about the distance to a scene with respect to the scene.SOLUTION: A depth image sensing device 1 comprises: a line memory 560 that detects dot pattern light reflected from a scene and stores the generated depth data in units of line groups L1 to L8; a dot center pixel detector 100 that determines a dot center pixel DCP having the highest intensity of depth data in each of a plurality of kernels included in the line groups L1 to L8; and a dot data memory 570 that stores dot center pixel information including a position of the dot center pixel DCP.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a depth image sensing device, an image signal processor and an image signal processing method capable of processing information relating to distance to a scene. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of light-sensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] Image sensing devices can be used to obtain a color image of a scene or to obtain a depth image of a scene. Image sensing devices capable of obtaining a depth image use separate light sources, but operating such light sources can consume significant power.

[0004] In particular, when an image sensing device is mounted on a mobile device, reducing the power required to operate the light source is an important issue. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical idea of ​​the present invention is to provide a depth image sensing device, an image signal processor, and an image signal processing method that can reduce the power required for operating a light source and improve the performance of depth sensing.

[0006] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] A depth image sensing device according to an embodiment of the present disclosure may include a line memory that senses dot pattern light reflected from a scene and stores the generated depth data in units of line groups, a dot center pixel detector that determines a dot center pixel having the strongest depth data intensity in each of a plurality of kernels included in the line group, and a dot data memory that stores dot center pixel information including the position of the dot center pixel.

[0008] An image signal processor according to another embodiment of the present disclosure may include a dot center pixel detector that determines a dot center pixel having the strongest depth data in each of a plurality of kernels, and a depth image generator that uses the depth data of the dot center pixel to calculate a distance to a scene.

[0009] An image signal processing method according to yet another embodiment of the present disclosure may include the steps of determining a dot center pixel having the strongest intensity of depth data in each of a plurality of kernels, comparing the intensity of the depth data of the dot center pixel with a significance threshold to determine whether the dot center pixel is a valid pixel, removing overlapping dot center pixels from among the dot center pixels that are valid pixels, and calculating a distance to a scene using the depth data of the dot center pixel. Effect of the Invention

[0010] According to the embodiments disclosed herein, a depth image sensing device, an image signal processor and an image signal processing method are provided that enable efficient and accurate distance detection independent of the scene.

[0011] In addition, various other effects that are directly or indirectly understood by this document may be provided. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating a schematic configuration of an imaging apparatus according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram for explaining depth data generated when the imaging device of FIG. 1 uses a direct ToF (time of flight) method. [Diagram 3] 2 is a diagram for explaining depth data generated when the imaging device of FIG. 1 uses an indirect ToF method. FIG. [Figure 4] FIG. 2 is a block diagram illustrating a dot center pixel detector according to one embodiment of the present invention. [Diagram 5] 5 is a flow chart illustrating a method of operation of the dot center pixel detector of FIG. 4. [Figure 6] FIG. 6 is a diagram for explaining step S20 in FIG. 5. [Figure 7] FIG. 6 is a diagram for explaining step S30 in FIG. 5. [Figure 8] FIG. 6 is a diagram for explaining step S50 in FIG. [Figure 9] FIG. 5 is a block diagram of one embodiment of an imaging device including the dot center pixel detector shown in FIG. 4. [Figure 10] 10 is a flow chart illustrating a method of operation of the imaging device shown in FIG. 9. [Figure 11] 11 is a flowchart showing step S130 shown in FIG. 10 in more detail. [Figure 12]11 is a flowchart showing step S150 shown in FIG. 10 in more detail. [Figure 13] 5 is a block diagram illustrating another embodiment of an imaging device including the dot center pixel detector shown in FIG. 4. [Figure 14] 14 is a flow chart illustrating a method of operation of the imaging device shown in FIG. 13. [Figure 15] FIG. 11 is a diagram for explaining an operation performed on a first line group. [Figure 16] FIG. 11 is a diagram for explaining an operation performed on a second line group. [Figure 17] FIG. 15 is a diagram for explaining step S250 in FIG. 14. [Figure 18] FIG. 15 is a diagram for explaining step S250 in FIG. 14. [Figure 19] 15 is a flowchart showing step S230 shown in FIG. 14 in more detail. [Figure 20] 15 is a flowchart showing step S250 shown in FIG. 14 in more detail. [Figure 21] 5 is a block diagram of an example computing device that corresponds to the dot center pixel detector of FIG. 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Various embodiments of the present invention will now be described with reference to the accompanying drawings, however, it should be understood that this is not intended to be limited to the specific embodiments, but rather includes various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0014] FIG. 1 is a diagram illustrating a schematic configuration of an imaging device according to an embodiment of the present invention.

[0015] 1, an imaging device 1 is a device for acquiring a depth image of a scene, and the depth image may be an image showing a sense of distance to the scene, that is, a user can visually confirm the distance to the scene through the depth image.

[0016] For acquiring a depth image, the imaging device 1 may include a light source module 2 and a camera 3 .

[0017] The light source module 2 emits light toward a scene, and the camera 3 senses the emitted light reflected from the scene and generates an electrical signal. A processor (not shown) provided inside or outside the camera 3 can generate a depth image based on the electrical signal generated by the camera 3.

[0018] The light source module 2 may consume a considerable amount of power to emit light, and in order to reduce such power consumption, the light source module 2 may emit dot pattern light to a scene. The dot pattern light may mean a plurality of lights emitted in a discontinuous form in a predetermined area, rather than a surface light, which is a single light emitted in a continuous form in a predetermined area. Thus, the dot pattern light may be displayed as a light having a dot shape in the scene and the camera 3.

[0019] In order to accurately detect the dot pattern light, it is necessary to accurately detect the center of each dot constituting the dot pattern light. However, since each dot of the dot pattern light incident on the camera 3 has scene-dependent characteristics, it may be difficult to accurately detect the center of each dot using a fixed method. The scene-dependent characteristics may occur due to a difference in position between the light source module 2 and the camera 3 (position deviation of the dot), or due to a difference in distance between the camera 3 and the scene (size deviation of the dot). This disclosure discloses a method that can accurately detect the dot pattern light despite the scene-dependent characteristics of each dot.

[0020] FIG. 2 is a diagram for explaining depth data generated when the imaging device of FIG. 1 uses a direct ToF (time of flight) method.

[0021] 2, the imaging device 1 may generate depth data that is the basis of a depth image using a direct ToF method, which may directly measure the time difference between the time when the modulated optical signal MLS is irradiated onto a scene and the time when the modulated optical signal MLS is reflected from the scene and incident thereon. In this case, the depth data may be pixel data obtained by sensing the modulated optical signal MLS reflected from the scene.

[0022] The camera 3 may include a plurality of direct pixels for the direct ToF method. According to an embodiment, each of the plurality of direct pixels may be a pixel including a single-photon avalanche diode (SPAD). When a reverse bias voltage is applied to the SPAD to increase the electric field, electrons generated by incident photons move due to the strong electric field, and impact ionization occurs, in which electron-hole pairs are generated. In particular, in a SPAD operating in a Geiger mode in which a reverse bias voltage higher than a breakdown voltage is applied, carriers (electrons or holes) generated by incident light and electrons and holes generated by the impact ionization phenomenon collide with each other to generate countless carriers. Therefore, even if a single photon is incident on the SPAD, the single photon triggers an avalanche breakdown, thereby generating a measurable current pulse.

[0023] In FIG. 2, the modulated optical signal MLS emitted from the light source module 2 may have a pulse shape with a relatively large amplitude and a small pulse width, and the time point at which the pulse occurs may be defined as a reference pulse time point RPT.

[0024] The light source module 2 irradiates a modulated light signal MLS to a scene, and the camera 3 detects the reflected light reflected from the scene and generates pixel data PD corresponding to the generated current pulse. The camera 3 can determine a time when the pixel data PD has a value equal to or greater than a threshold data as a pulse detection time PST, calculate a time of flight ToF that is a difference in time from a reference pulse time RPT to the pulse detection time PST, and calculate the calculated time of flight ToF and the speed of light (e.g., multiply a value obtained by dividing ToF by 2 by the speed of light) to calculate a distance between the scene and the camera 3.

[0025] FIG. 3 is a diagram for explaining depth data generated when the imaging device of FIG. 1 uses the indirect ToF method.

[0026] 3, the imaging device 1 can generate depth data that is the basis of a depth image using an indirect ToF method, which can be a method of calculating a phase difference between a modulated optical signal MLS irradiated to a scene and a reflected modulated optical signal MLS_R that is reflected from the scene and incident on the scene. In this case, the depth data can be pixel data corresponding to photocharges captured by pixels controlled by a modulated control signal having a predetermined phase difference (e.g., 0 degrees, 90 degrees, 180 degrees, or 270 degrees) with the modulated optical signal MLS.

[0027] The camera 3 may include a plurality of indirect pixels for the indirect ToF method. According to an embodiment, each of the plurality of indirect pixels may be a pixel having a 2-tap structure. The pixel having the 2-tap structure may include one photoelectric conversion element that converts incident light into photoelectric charges, two transistors that are turned on by first and second modulation control signals MCS1 and MCS2 having opposite phases to each other and transmit the photoelectric charges at different timings, and two output circuits that convert the photoelectric charges output from the two transistors into electrical signals and output them. For example, the first modulation control signal MCS1 and the second modulation control signal MCS2 may have a phase difference of 0 degrees (i.e., in phase) and a phase difference of 180 degrees with the modulated optical signal MLS, respectively. Or, the first modulation control signal MCS1 and the second modulation control signal MCS2 may have a phase difference of 90 degrees and a phase difference of 270 degrees with the modulated optical signal MLS, respectively.

[0028] The modulated optical signal MLS shown in Fig. 3 may have pulses with a relatively smaller amplitude and a larger pulse width than the modulated optical signal MLS shown in Fig. 2. The reflected modulated optical signal MLS_R may have a predetermined phase difference θ with the modulated optical signal MLS depending on the flight time of the modulated optical signal MLS reflected by the scene and incident thereon. The operation period for the camera 3 to calculate the distance to the scene is divided into a first sensing period SP1 and a second sensing period SP2. It is also assumed that the phase difference θ between the modulated optical signal MLS and the reflected modulated optical signal MLS_R is maintained constant in the first sensing period SP1 and the second sensing period SP2.

[0029] In the first sensing period SP1, the first modulation control signal MCS1 may have the same phase (phase difference of 0 degrees) as the modulated optical signal MLS and may have an activation voltage (or logic high level) in the first section PR1. First depth data C0 indicating the amount of photocharges generated in the first section PR1 may be generated by the first modulation control signal MCS1 having the activation voltage.

[0030] In the first sensing period SP1, the second modulation control signal MCS2 has an opposite phase (a phase difference of 180 degrees) to the modulated optical signal MLS and may have an activation voltage (or a logic high level) in the second section PR2. Third depth data C2 indicating the amount of photocharges generated in the second section PR2 may be generated by the second modulation control signal MCS2 having the activation voltage.

[0031] In the second sensing period SP2, the first modulation control signal MCS1 has a phase difference of 90 degrees with the modulated optical signal MLS and may have an activation voltage (or a logic high level) in the third section PR3. Second depth data C1 indicating the amount of photocharges generated in the third section PR3 may be generated by the first modulation control signal MCS1 having the activation voltage.

[0032] In the second sensing period SP2, the second modulation control signal MCS2 has a phase difference of 270 degrees with the modulated optical signal MLS and may have an activation voltage (or a logic high level) in the fourth section PR4. Fourth depth data C3 indicating the amount of photocharges generated in the fourth section PR4 may be generated by the second modulation control signal MCS2 having the activation voltage.

[0033] That is, the reflected modulated optical signal MLS_R can be captured in the first sensing period SP1 divided into a first section PR1 and a second section PR2, and in the second sensing period SP2 divided into a third section PR3 and a fourth section PR4.

[0034] The phase difference θ between the modulated optical signal MLS and the reflected modulated optical signal MLS_R can be calculated from the first to fourth depth data C0 to C3 by the following mathematical formula 1.

[0035] [Mathematical formula 1]

number

[0036] In addition, the distance d between the camera 3 and the scene can be calculated using the phase difference θ according to the following mathematical formula 2.

[0037] [Mathematical formula 2]

number

[0038] Here, c may be the speed of light and fm may be the frequency of the modulated optical signal MLS.

[0039] On the other hand, the amplitude value (amp) of the reflected modulated optical signal MLS_R can be calculated by the following mathematical formula 3.

[0040] [Mathematical formula 3]

number

[0041] In addition, the charge value (B), which indicates the total charge of the photocharges generated by the reflected modulated optical signal MLS_R, can correspond to the sum C0+C2 of the first depth data C0 and the third depth data C2 or the sum C1+C3 of the second depth data C1 and the fourth depth data C3.

[0042] According to the phase difference detection method (4-phase modulation method) as shown in FIG. 3, by using differential values ​​such as (C1-C3) and (C0-C2) when calculating the phase difference, background noise components included in each of the first to fourth depth data C0-C3 can be removed, making it possible to calculate the distance more accurately.

[0043] Fig. 4 is a block diagram showing a dot center pixel detector according to an embodiment of the present invention. Fig. 5 is a flow chart for explaining a method of operation of the dot center pixel detector of Fig. 4. Fig. 6 is a diagram for explaining step S20 of Fig. 5. Fig. 7 is a diagram for explaining step S30 of Fig. 5. Fig. 8 is a diagram for explaining step S50 of Fig. 5.

[0044] Referring to Figures 4 and 5, the dot center pixel detector 100 can be a configuration included in the camera 3 of Figure 1, but the scope of the present invention is not limited thereto and can be a configuration located outside the camera 3.

[0045] The dot center pixel detector 100 can receive and analyze depth data to determine a dot center pixel from the depth data, and generate and output dot center pixel information. The camera 3 can include a pixel array in which a plurality of pixels are arranged in a matrix form including a plurality of rows and a plurality of columns to obtain depth data for a scene. A plurality of pixels constituting the pixel array can generate depth data for one scene, and such depth data can constitute one frame. The depth data input to the dot center pixel detector 100 can correspond to one frame, and the depth data corresponding to one frame can be input sequentially through a serial interface rather than being input simultaneously.

[0046] Since the reflected modulated light signal input to the camera 3 corresponds to a dot pattern light, there may be a region where light is incident (e.g., DOT in FIG. 6) and a region where light is not incident (e.g., a region other than DOT in FIG. 6) in one frame. It is meaningless to calculate the distance to the scene based on the depth data corresponding to the region where light is not incident, and the intensity of the depth data at the dot center pixel located at the center of the region where light is incident may be the highest even in the region where light is incident, and the intensity of the depth data may gradually decrease toward the periphery of the dot center pixel. The higher the intensity of the depth data, the better the signal-to-noise ratio (SNR) may be, and it may be most effective to calculate the distance to the scene using the depth data of the dot center pixel. Therefore, the operation of determining the dot center pixel by the dot center pixel detector 100 may be important in the process of distance detection using the dot pattern light.

[0047] The dot center pixel information may include a position of the dot center pixel and depth data of the dot center pixel. According to an embodiment, the dot center pixel information may further include data in which the depth data of the dot center pixel is processed.

[0048] The dot center pixel detector 100 may include a search point management unit 110, a nearby pixel comparison unit 120, a valid pixel determination unit 130, and an overlapped dot center pixel removing unit 140. The dot center pixel detector 100 may be implemented in hardware, software, and / or a combination thereof.

[0049] The search point management unit 110 can acquire depth data corresponding to one frame (S10). Here, the description is given on the premise that the search point management unit 110 acquires depth data in units of one frame, but in other embodiments, the search point management unit 110 can acquire depth data in units smaller than one frame (for example, depth data corresponding to the capacity of a line memory).

[0050] In Fig. 6, depth data corresponding to one frame is illustrated, and the depth data corresponding to the dot pattern light is displayed as dots DOT. The brightness in the dots DOT can indicate the intensity of the depth data, and the lower the brightness (i.e., the darker it is), the stronger the intensity of the depth data can be. In Fig. 6, the number and positions of the dots DOT and the number and positions of the search points SP are merely exemplary for the convenience of explanation, and the scope of the present invention is not limited thereto.

[0051] In the case where each pixel is a direct pixel, the intensity of the depth data may be pixel data obtained by sensing the modulated light signal MLS reflected from the scene. In another embodiment, in the case where each pixel is an indirect pixel, the intensity of the depth data may be an amplitude value or a charge value calculated from pixel data (i.e., the first to fourth depth data C0 to C3) corresponding to photocharges captured by pixels controlled by a modulated control signal having a predetermined phase difference (e.g., 0 degree, 90 degree, 180 degree, or 270 degree) with the modulated light signal MLS.

[0052] The search point management unit 110 may determine or manage the initial positions of search points arranged according to a predetermined rule for depth data corresponding to one frame (S20). Here, the search point may refer to the position of a pixel that is a unit for searching for a dot center pixel. In another embodiment, if the search point is fixed in advance, step S20 may be omitted.

[0053] In FIG. 6, the search points SP may be arranged to correspond to the arrangement of dots constituting the dot pattern light, and the search points SP may be arranged at each intersection of the grid to correspond to the dots arranged in a row along the horizontal and vertical directions. According to an embodiment, the interval between adjacent search points SP in the horizontal direction may be set smaller than the interval between adjacent dots DOT in the horizontal direction, and the interval between adjacent search points SP in the vertical direction may be set smaller than the interval between adjacent dots DOT in the vertical direction. This is to prevent the existence of a dot center pixel (i.e., a missed pixel) that is not searched by any search point. Here, the interval between the dots DOT may be experimentally determined as the minimum interval between the dots DOT. The arrangement of the search points SP is not limited to FIG. 6, and may be arranged in a zigzag pattern, for example.

[0054] The search point management unit 110 may determine the number and positions of the search points SP in a predetermined manner according to the arrangement of the dots constituting the dot pattern light, and store the positions of the search points SP. In another embodiment, the search point management unit 110 may dynamically adjust the positions and / or number of the pre-stored search points SP. For example, when the number of dot center pixels included in the dot center pixel information is equal to or less than a predetermined ratio (e.g., 50%) of the number of the total dots DOT, the search point management unit 110 may increase the number of search points SP and reduce the interval between adjacent search points SP.

[0055] The surrounding pixel comparator 120 may determine a dot center pixel DCP for each search point SP based on depth data of pixels (i.e., surrounding pixels) located within a predetermined range of the search point SP. According to an embodiment, the surrounding pixel comparator 120 may calculate the intensity of depth data of the surrounding pixels and compare the intensity of the depth data to determine the dot center pixel DCP. That is, the surrounding pixel comparator 120 may determine a pixel having the strongest intensity of depth data among the surrounding pixels of the search point SP as the dot center pixel DCP (S30).

[0056] In Fig. 6, a set of pixels located within a predetermined range for each search point SP can be defined as a kernel (KN). According to one embodiment, the horizontal length of the kernel can be set to be greater than the interval between horizontally adjacent search points SP and less than twice the interval, and the vertical length of the kernel can be set to be greater than the interval between vertically adjacent search points SP and less than twice the interval. This is merely an example, and the size of the kernel KN can be experimentally determined so as to minimize unnecessary calculations (duplicate detection of dot center pixels DCP) and non-detection of dot center pixels DCP.

[0057] The valid pixel determination unit 130 may compare the intensity of depth data of each dot center pixel DCP determined by the surrounding pixel comparison unit 120 with a valid threshold value to determine whether each dot center pixel DCP is a valid pixel (S40). The valid threshold may be a fixed value that is predetermined in consideration of the maximum and minimum values ​​of the intensity of the depth data, the signal-to-noise ratio, etc. According to another embodiment, if the number of dot center pixels DCP included in the dot center pixel information is less than a predetermined ratio (e.g., 50%) of the number of all dots DOT, the valid pixel determination unit 130 may reduce the magnitude of the valid threshold. In still another embodiment, when the number of dot center pixels DCP is less than a predetermined ratio (e.g., 50%) of the total number of dots DOT, steps S30 and S40 can be repeated after expanding a predetermined range of the search points SP for determining the dot center pixels DCP for each search point SP in the surrounding pixel comparator 120. This allows the dot center pixels DCP to be detected more accurately while minimizing resources required for determining the dot center pixels DCP.

[0058] 6 and 7, arrows connecting dot center pixels DCP determined by a kernel KN including each search point SP and the search point SP are displayed.

[0059] If the intensity of the depth data of the dot center pixel DCP of the search point SP is equal to or greater than the validity threshold, the valid pixel determination unit 130 can determine the search point SP as a valid search point SP_V. Also, the dot center pixel DCP corresponding to the valid search point SP_V can be determined as a valid pixel.

[0060] If the intensity of the depth data of the dot center pixel DCP of the search point SP is less than the validity threshold, the valid pixel determination unit 130 can determine the search point SP_I as an invalid search point SP_I. Also, the dot center pixel corresponding to the invalid search point SP_I can be determined as an invalid pixel.

[0061] The overlapping dot center pixel removal unit 140 can remove overlapping dot center pixels from among the dot center pixels DCP that are valid pixels (S50).

[0062] In Fig. 8, the overlapping dot center pixel DCP_O (shown as a double circle in Fig. 8) may refer to a pixel corresponding to a plurality of valid search points SP_V. The normal dot center pixel DCP_N may refer to a pixel corresponding to one valid search point SP_V. If information on a dot center pixel DCP that is overlapping and substantially identical to the overlapping dot center pixel DCP_O is included in the dot center pixel information, the distance to the scene may be calculated unnecessarily multiple times, so the overlapping dot center pixel remover 140 may select only one dot center pixel from the overlapping dot center pixels DCP_O.

[0063] Here, the overlapping dot center pixel removal unit 140 may determine that the dot center pixels DCP_O are overlapping if the positions of the dot center pixels DCP are the same or located within a predetermined distance (e.g., the diameter of a dot of a dot pattern light). According to one embodiment, if the positions of the dot center pixels DCP are the same, the overlapping dot center pixel removal unit 140 may select only the dot center pixel DCP corresponding to the search point SP having the smallest or largest index (i.e., identification number) of the search point SP. According to another embodiment, if the positions of the dot center pixels DCP are located within a predetermined distance, the overlapping dot center pixel removal unit 140 may select only the dot center pixel DCP having the largest intensity of depth data.

[0064] The dot center pixel detector 100 of the present disclosure can accurately detect dot center pixels despite scene-dependent characteristics or other distortions (e.g., lens distortion, variations in the relative positional relationship between the light source 2 and the camera 3), accurately calculate the distance to the scene, and minimize unnecessary calculations.

[0065] FIG. 9 is a block diagram of one embodiment of an imaging device including the dot center pixel detector shown in FIG.

[0066] Referring to FIG. 9, the imaging device 900 may include a depth image sensing device 200 , an image signal processor 300 , and a frame memory 400 .

[0067] The depth image sensing device 200 may generate depth data for a target object OBJ using at least one depth sensing technology. The depth image sensing device 200 may include a light source module 210, a pixel array 220, a pixel driver 230, a timing controller 240, and a pixel readout circuit 250.

[0068] The light source module 210 can irradiate the target object OBJ with a modulated light signal MLS in response to a control signal from the timing controller 240. The light source module 210 can be a laser diode (LD) or a light emitting diode (LED) that emits light in a specific wavelength band (e.g., infrared light or visible light), a near infrared laser (NIR), a point light source, a monochromatic illumination source in which a white lamp and a monochromator are combined, or a combination of other laser light sources. For example, the light source module 210 can emit infrared light having a wavelength of 800 nm to 1000 nm. The modulated light signal MLS can be an optical pulse signal modulated with predetermined modulation characteristics (e.g., waveform, wavelength, period, amplitude, frequency, phase, duty ratio, etc.).

[0069] The pixel array 220 may include a plurality of pixels arranged in a two-dimensional matrix structure (e.g., arranged in a row direction and a column direction). In the embodiments shown in FIG. 9 and the following figures, each pixel may be a direct pixel or an indirect pixel. For convenience of explanation, the following description is based on the assumption that each pixel is an indirect pixel.

[0070] Each pixel of the pixel array 220 can perform photoelectric conversion of the reflected modulated optical signal MLS_R received through a lens module (not shown) under the control of the pixel driver 230, generate a pixel signal which is an electrical signal corresponding to the intensity of the reflected modulated optical signal MLS_R, and output it to the pixel readout circuit 250.

[0071] The pixel driver 230 can generate and supply to the pixel array 220 signals for controlling each pixel of the pixel array 220 under the control of the timing controller 240. In particular, the pixel driver 230 can generate and supply to the pixel array 220 a first modulation control signal MCS1 and a second modulation control signal MCS2 for controlling the timing at which each tab captures a photocharge.

[0072] The timing controller 240 may control the overall operation of the depth image sensing device 200 by controlling the light source module 210 , the pixel driver 230 and the pixel readout circuit 250 .

[0073] The pixel readout circuit 250 may process analog pixel signals output from each pixel to generate depth data, which is digital data corresponding to the pixel signals. For example, the pixel readout circuit 250 may include an analog-to-digital converter for analog-to-digital conversion of the pixel signals into depth data.

[0074] The image signal processor 300 may collect depth data input from the depth image sensing device 200 and generate a depth image indicating a distance to a target object OBJ. The image signal processor 300 may also perform image signal processing for noise removal and image quality improvement on the generated depth image. The depth image output from the image signal processor 300 may be stored in an internal memory or an external memory of the imaging device 900 or an apparatus in which the imaging device 900 is installed, or may be displayed via a display, depending on a user's request or automatically. Alternatively, the depth image output from the image signal processor 300 may be used to control the operation of the imaging device 900 or an apparatus in which the imaging device 900 is installed.

[0075] The image signal processor 300 may include a dot center pixel detector 100. The image signal processor 300 may store depth data input from the depth image sensing device 200 in a frame memory 400, and then provide depth data corresponding to one frame to the dot center pixel detector 100. The dot center pixel detector 100 may generate dot center pixel information, which is information on a plurality of dot center pixels, for the depth data corresponding to one frame.

[0076] The depth image generator 310 calculates the phase difference between the modulated light signal MLS and the reflected modulated light signal MLS for each of the multiple dot center pixels by referring to the dot center pixel information, and calculates the distance between the target object OBJ and the depth image sensing device 200 based on the calculated phase difference to generate a depth image.

[0077] The frame memory 400 may receive and store the depth data generated by the depth image sensing device 200 via the image signal processor 300. The frame memory 400 may have a storage capacity capable of storing depth data corresponding to one frame. The frame memory 400 may also store data (e.g., information on search points, dot center pixel information, etc.) and instructions required for the operation of the image signal processor 300.

[0078] The imaging device 900 may correspond to a specific implementation of the imaging device 1 described in FIG. 1, and may be an embodiment in which the dot center pixel detector 100 is included in the image signal processor 300.

[0079] FIG. 10 is a flow chart illustrating a method of operation of the imaging device shown in FIG.

[0080] Referring to FIG. 10, the operation method of the imaging device shown in FIG. 10 is substantially the same as the operation method of the dot center pixel detector described in FIG. 5 except for some operations, so the following description will focus on the differences.

[0081] First, the depth image sensing device 200 may generate first depth data C0 and third depth data C2 during a first sensing period SP1, and the image signal processor 300 may store the first depth data C0 and third depth data C2 in the frame memory 400 (S110).

[0082] The depth image sensing device 200 may generate second depth data C1 and fourth depth data C3 during the second sensing period SP2, and the image signal processor 300 may store the second depth data C1 and the fourth depth data C3 in the frame memory 400 (S112).

[0083] The search point management unit 110 can determine or manage the initial positions of the search points for the first to fourth depth data C0 to C3 corresponding to one frame according to a predetermined rule (S120).

[0084] The surrounding pixel comparator 120 can determine a dot center pixel DCP for each search point SP based on depth data of pixels located within a predetermined range of the corresponding search point SP (S130).

[0085] The valid pixel determination unit 130 can compare the intensity of the depth data of each dot center pixel DCP determined by the surrounding pixel comparison unit 120 with a validity threshold to determine whether each dot center pixel DCP is a valid pixel (S140). According to an embodiment, when the number of dot center pixels DCP is less than a predetermined ratio (e.g., 50%) of the total number of dots DOT, the surrounding pixel comparator 120 may expand a predetermined range of the search points SP for determining the dot center pixel DCP for each search point SP, and then repeat steps S130 and S140. This allows the dot center pixel DCP to be detected more accurately while minimizing resources required for determining the dot center pixel DCP.

[0086] The overlapping dot center pixel removal unit 140 can remove overlapping dot center pixels from among the dot center pixels DCP that are effective pixels (S150).

[0087] FIG. 11 is a flowchart showing step S130 shown in FIG. 10 in more detail.

[0088] Referring to Fig. 11, the surrounding pixel comparator 120 may initialize (i=1) a search point index i, which is identification information of a search point SP (S131). Each of the search points SP shown in Fig. 6 may have a different search point index, for example, the search point index of the search point SP located at the leftmost position of the first row in Fig. 6 may be 1, and the search point index may increase by 1 toward the right, and the search point index of the search point SP located at the leftmost position of the second row may be 12, and the search point index may increase by 1 toward the right. It is assumed that search point indexes are assigned to a total of 77 search points SP in this manner.

[0089] The surrounding pixel comparator 120 detects the position X of the search point SP corresponding to the current search point index. i , Y i and obtain the amplitude value amp of the search point SP. i The amplitude value amp can be calculated (S132). i The calculation of can be performed by using the first to fourth depth data C0 to C3 for the search point SP through Equation 3. Also, the surrounding pixel comparator 120 calculates the position X of the search point SP by using the first to fourth depth data C0 to C3 for the search point SP through Equation 3. i , Y i and the amplitude value amp i can be updated to the position of the dot center pixel and the amplitude value of the dot center pixel, respectively.

[0090] The surrounding pixel comparison unit 120 compares an arbitrary pixel X i+△x , Y i+△y The first to fourth depth data C0 to C3 can be obtained (S133).

[0091] The surrounding pixel comparison unit 120 compares pixel X i+△x , Y i+△y Using the first to fourth depth data C0 to C3, pixel Xi+△x , Y i+△y The amplitude value amp_cal can be calculated (S134).

[0092] The calculated amplitude value amp_cal is the amplitude value amp of the dot center pixel. i If it is greater than the pixel X (Yes in S135), the surrounding pixel comparison unit 120 i+△x , Y i+△y The position of the dot and the calculated amplitude value amp_cal are calculated by the position of the center pixel X i , Y i and the amplitude value of the dot center pixel amp i (S136).

[0093] The calculated amplitude value amp_cal is the amplitude value amp of the dot center pixel. i If it is equal to or smaller than the above (No in S135), step S136 is not performed, and the position X of the dot center pixel is i , Y i and the amplitude value amp i can be maintained.

[0094] The surrounding pixel comparison unit 120 can determine whether step S133 has been performed for all pixels in the kernel KN including the search point SP (S137).

[0095] If step S133 has not been performed for all pixels in the kernel KN (No in S137), the step S133 is performed for any pixel X i+△x , Y i+△y Step S133 may be performed for other pixels in the kernel other than KN. Then, steps S133 to S137 may be performed repeatedly for all pixels in the kernel. When steps S133 to S137 have been performed for all pixels in the kernel KN, the highest amplitude value amp i The pixel having the position X of the dot center pixel is determined as the dot center pixel. i , Yi and the amplitude value amp i can be stored in the frame memory 400.

[0096] When step S133 has been performed for all pixels in the kernel KN (Yes in S137), if step S132 has not been performed for all search points (No in S138), the surrounding pixel comparator 120 increments the current search point index i by 1 (S139) and can perform step S132 again. This operation can be repeated until step S132 has been performed for all search points. When step S132 has been performed for all search points (Yes in S138), the position X of the dot center pixel is calculated for each of the search points. i , Y i and the amplitude value amp i can be stored in the frame memory 400.

[0097] FIG. 12 is a flowchart showing step S150 shown in FIG. 10 in more detail.

[0098] 12, the overlapped dot center pixel remover 140 can initialize (i=1) a search point index i, which is identification information of a search point SP (S151). Here, it should be noted that each operation in FIG. 12 can be performed only for a valid search point SP_V determined in step S140, and the search point SP referred to in FIG. 12 is a valid search point SP_V.

[0099] The overlapped dot center pixel remover 140 detects the position X of the dot center pixel corresponding to the search point SP that corresponds to the current search point index. i , Y i can be obtained (S152).

[0100] The overlapped dot center pixel removal unit 140 detects the position X of the dot center pixel corresponding to the adjacent search point SP corresponding to the current search point index. j , Y j Here, the adjacent search point may be a search point adjacent to the search point SP corresponding to the current search point index in the upper, lower, left, right, or diagonal direction, and may refer to a search point where the search point SP corresponding to the current search point index and the kernel KN overlap.

[0101] The overlapped dot center pixel remover 140 detects the position X of the dot center pixel corresponding to the search point SP that corresponds to the current search point index. i , Y i and the position X of the dot center pixel corresponding to the adjacent search point. j , Y j can be compared to determine whether the two positions are identical to each other (S154).

[0102] Dot center pixel position X i , Y i and the position of the dot center pixel X j , Y j If they are the same (Yes in S154), the dot center pixels overlap each other, so the overlapping dot center pixel remover 140 can remove one of the overlapping dot center pixels (S155). According to an embodiment, the overlapping dot center pixel remover 140 can select only the dot center pixel DCP corresponding to the search point SP whose index is relatively smaller or larger than the search point SP.

[0103] Dot center pixel position X i , Y i and the position of the dot center pixel X j , Y jIf they are not identical to each other (No in S154), the step S155 may not be performed since the dot center pixels do not overlap each other.

[0104] The overlapping dot center pixel remover 140 can determine whether the step S153 has been performed for all adjacent search points to the search point SP (S156).

[0105] If step S153 has not been performed for all adjacent search points for search point SP (No in S156), step S153 may be performed for other adjacent search points for search point SP. Thereafter, steps S153 through S156 may be repeatedly performed for other adjacent search points for search point SP, and when steps S153 through S156 have been performed for all adjacent search points for search point SP, overlapped dot center pixel removal unit 140 may remove one of the overlapped dot center pixels associated with search point SP.

[0106] When step S153 has been performed for all adjacent search points for search point SP (Yes in S156), if step S152 has not been performed for all search points SP (No in S157), the overlapping dot center pixel remover 140 increments the current search point index i by 1 (S158) and can perform step S152 again. This operation can be repeated until step S152 has been performed for all search points. When step S152 has been performed for all search points (Yes in S157), the position X of the dot center pixel is i , Y i and the amplitude value amp i The dot center pixel information for all the dot center pixels can be stored in the frame memory 400 so that they do not overlap.

[0107] When determining the dot center pixel by using the amplitude value as the intensity of the depth data as in the embodiment described in Figures 9 to 12, only the active components (actual signal components) other than the background light components can be used, and the dot center pixel can be determined more accurately.

[0108] FIG. 13 is a block diagram illustrating another embodiment of an imaging device including the dot center pixel detector shown in FIG.

[0109] Referring to FIG. 13, the imaging device 1300 may include a depth image sensing device 500 and an image signal processor 600 .

[0110] The imaging device 1300 may not include a frame memory for the operation of the dot center pixel detector 100, the dot center pixel detector 100 may be included in the depth image sensing device 500 rather than the image signal processor 600, and a line memory 560 and a dot data memory 570 for the operation of the dot center pixel detector 100 may be included in the depth image sensing device 500.

[0111] The depth image sensing device 500 may include a light source module 510, a pixel array 520, a pixel driver 530, a timing controller 540, a pixel readout circuit 550, a line memory 560, a dot data memory 570, and a dot center pixel detector 100. The functions and operations of the light source module 510, the pixel array 520, the pixel driver 530, the timing controller 540, and the pixel readout circuit 550 are substantially the same as those of the light source module 510, the pixel array 520, the pixel driver 530, the timing controller 540, and the pixel readout circuit 550 described in FIG. 9, so that the duplicated description will be omitted.

[0112] The line memory 560 may store depth data output from the pixel read circuit 250. The line memory 560 may have a capacity for storing depth data corresponding to a portion of one frame. Here, the portion of one frame may correspond to pixels included in a plurality of rows of the pixel array 520, and the plurality may be defined to be equal to or greater than the interval between adjacent search points SP in the vertical direction, but the scope of the present invention is not limited thereto.

[0113] The dot data memory 570 can store data (eg, information about search points, dot center pixel information, etc.) and instructions required for the operation of the dot center pixel detector 100 .

[0114] By including the dot center pixel detector 100 in the depth image sensing device 500, the line memory 560 and dot data memory 570 installed in the depth image sensing device 500 can be realized with a minimum capacity, thereby minimizing the required memory capacity.

[0115] The dot center pixel detector 100 can generate dot center pixel information using only resources within the depth image sensing device 500. The depth image generator 610 of the image signal processor 600 can refer to the dot center pixel information transmitted from the depth image sensing device 500, calculate a phase difference between the modulated optical signal MLS and the reflected modulated optical signal MLS for each of the plurality of dot center pixels, and calculate a distance between the target object OBJ and the depth image sensing device 200 based on the calculated phase difference to generate a depth image.

[0116] Fig. 14 is a flow chart for explaining an operation method of the imaging device shown in Fig. 13. Fig. 15 is a diagram for explaining an operation performed on the first line group. Fig. 16 is a diagram for explaining an operation performed on the second line group. Figs. 17 and 18 are diagrams for explaining step S250 in Fig. 14.

[0117] Referring to FIG. 14, the operation method of the imaging device shown in FIG. 14 is substantially the same as the operation method of the dot center pixel detector described in FIG. 5 except for some operations, so the following description will focus on the differences.

[0118] First, the depth image sensing device 500 may generate first depth data C0 and third depth data C2 in a first sensing period SP1 and store the first depth data C0 and third depth data C2 of a current line (i.e., row) of the pixel array 520 in the line memory 560 (S210).

[0119] If the capacity of the line memory 560 is not full (No in S212), the depth image sensing device 500 can repeat step S210.

[0120] If the capacity of the line memory 560 is full (Yes in S212), the search point management unit 110 can determine or manage search points arranged according to a predetermined rule for the first and third depth data C0, C2 corresponding to a line group composed of lines stored in the line memory 560 (S220).

[0121] The surrounding pixel comparator 120 can determine a dot center pixel DCP for each search point SP based on depth data of pixels located within a predetermined range of the corresponding search point SP (S230).

[0122] The valid pixel determination unit 130 can compare the intensity of the depth data of each dot center pixel DCP determined by the surrounding pixel comparison unit 120 with a validity threshold to determine whether each dot center pixel DCP is a valid pixel (S240).

[0123] In FIG. 15, the surrounding pixel comparison unit 120 can determine the dot center pixel DCP1 of the dot DOT based on the depth data of pixels located within the kernel KN1 corresponding to each search point included in the first line group L1 stored in the line memory 560.

[0124] In addition, the valid pixel determination unit 130 can compare the intensity of the depth data of each dot center pixel DCP1 determined by the surrounding pixel comparison unit 120 with a valid threshold value to determine whether each dot center pixel DCP1 is a valid pixel. If the dot center pixel DCP1 is a valid pixel, the search point corresponding to the dot center pixel DCP1 can be determined as a valid search point SP_V. If the dot center pixel DCP1 is not a valid pixel, the search point corresponding to the dot center pixel DCP1 can be determined as an invalid search point SP_I.

[0125] After completing step S240, the dot center pixel detector 100 can determine whether the dot center pixel detection operation is completed for all lines of the pixel array 520 (S214).

[0126] If the dot center pixel detection operation is not completed for all the lines of the pixel array 520 (No in S214), the dot center pixel detector 100 may request the timing controller 540 to read and store the next line group. The timing controller 540 may control the pixel driver 530, the pixel read circuit 550, and the line memory 560 to perform step S210 for the next line group. Thus, steps S210 to S240 may be performed for the second line group L2, which is the next line group.

[0127] In FIG. 16, the surrounding pixel comparison unit 120 can determine the dot center pixel DCP2 of the dot DOT based on the depth data of pixels located within the kernel KN2 corresponding to each search point included in the second line group L2 stored in the line memory 560.

[0128] In addition, the valid pixel determination unit 130 can compare the intensity of the depth data of each dot center pixel DCP2 determined by the surrounding pixel comparison unit 120 with a valid threshold value to determine whether each dot center pixel DCP2 is a valid pixel. If the dot center pixel DCP2 is a valid pixel, the search point corresponding to the dot center pixel DCP2 can be determined as a valid search point SP_V. If the dot center pixel DCP2 is not a valid pixel, the search point corresponding to the dot center pixel DCP2 can be determined as an invalid search point SP_I.

[0129] When the dot center pixel detection operation is completed for all lines of the pixel array 520 (Yes in S214), the overlapping dot center pixel removal unit 140 can remove overlapping dot center pixels from among the dot center pixels that are valid pixels (S250).

[0130] 17, it is assumed that there are eight line groups L1 to L8 corresponding to the pixel array 520, but the scope of the present invention is not limited to this. A plurality of dot center pixels DCP1, DCP2 may correspond to each dot DOT, and the positions of the dot center pixels DCP1, DCP2 may differ. This is because the kernel size of each search point may not be sufficiently secured due to the capacity limit of the line memory 560.

[0131] 18, the overlapping dot center pixel remover 140 can remove the overlapping dot center pixel DCP2 from among the dot center pixels DCP1 and DCP2, which are effective pixels. Here, the overlapping dot center pixel remover 140 can remove the dot center pixel DCP2, which has a relatively small depth data intensity, from among the dot center pixels DCP1 and DCP2.

[0132] FIG. 19 is a flowchart showing step S230 shown in FIG. 14 in more detail.

[0133] Referring to FIG. 19, the surrounding pixel comparison unit 120 can initialize (i=1) a search point index i, which is identification information of a search point SP (S231).

[0134] The surrounding pixel comparator 120 detects the position X of the search point SP corresponding to the current search point index. i , Y i and obtain the charge value B of the search point SP. i The charge value Bi can be calculated as the sum C0+C2 of the first depth data C0 and the third depth data C2 for the search point SP (S232). The surrounding pixel comparator 120 also calculates the position X i , Y i and the charge value Bi can be updated to the position of the dot center pixel and the charge value of the dot center pixel, respectively.

[0135] The surrounding pixel comparison unit 120 compares an arbitrary pixel X i+△x , Y i+△y The first depth data C0 and the third depth data C2 can be obtained (S233).

[0136] The surrounding pixel comparison unit 120 compares pixel X i+△x , Y i+△y Using the first depth data C0 and the third depth data C2 of pixel X i+△x , Y i+△y The charge value B_cal can be calculated (S234). If the difference obtained by subtracting the average charge value Bave for the search point SP from the calculated charge value B_cal is greater than the threshold difference Dth (Yes in S235-1), the surrounding pixel comparator 120 regards the calculated charge value B_cal as an effective charge value and compares the calculated charge value B_cal with the charge value B of the dot center pixel. i can be compared (S235-2). If the value obtained by subtracting the average charge value Bave for the search point SP from the calculated charge value B_cal is less than or equal to the threshold difference Dth (No in S235-1), the surrounding pixel comparison unit 120 can consider the calculated charge value B_cal to be an invalid charge value and perform step S233 again. Here, the average charge value B ave is the charge value B of pixels located within a predetermined range around the search point SP (i.e., surrounding pixels). i The threshold difference Dth may be an experimentally predetermined value. i contains not only the component of the reflected modulated optical signal MLS_R but also the component of the background light (ambient light), so the charge value B i is not used as a reference value as is, but the calculated charge value B_cal is considered to be a valid value only when the difference between the calculated charge value B_cal and the average charge value Bave is greater than a predetermined threshold difference Dth, thereby making it possible to update only the component due to the dot light, not the component due to the background light, in the dot center pixel.

[0137] The calculated charge value B_cal is the charge value B of the dot center pixel. i If it is greater (Yes in S235-2), the surrounding pixel comparison unit 120 i+△x , Y i+△y The position of the dot and the calculated charge value B_cal are calculated by the position of the dot center pixel X i , Y i and the charge value of the dot center pixel B i (S236).

[0138] The calculated charge value B_cal is the charge value B of the dot center pixel. i If it is equal to or less than the above (No in S235-2), step S236 is not performed, and the position X of the dot center pixel is i , Y i and the charge value B i can be maintained.

[0139] The surrounding pixel comparison unit 120 can determine whether or not step S233 has been performed for all pixels in the kernel including the search point SP (S237).

[0140] If step S233 has not been performed for all pixels in the kernel (No in S237), the step S233 is performed for any pixel X i+△x , Y i+△y Step S233 may be performed for other pixels in the kernel other than the pixel 1. Thereafter, steps S233 to S237 may be repeatedly performed for all pixels in the kernel. When steps S233 to S237 have been performed for all pixels in the kernel, the highest charge value B i The pixel having the position X of the dot center pixel is determined as the dot center pixel. i , Y i and the charge value B i can be stored in the dot data memory 570.

[0141] When step S233 has been performed for all pixels in the kernel (Yes in S237), if step S232 has not been performed for all search points (No in S238), the surrounding pixel comparator 120 increments the current search point index i by 1 (S239) and can perform step S232 again. This operation can be repeated until step S232 has been performed for all search points. When step S232 has been performed for all search points (Yes in S238), the position X of the dot center pixel is calculated for each of the search points. i , Y i and the charge value B i can be stored in the dot data memory 570. In FIG. 19, when a line memory 560 is used instead of the frame memory 400, the charge value B i In the above embodiment, necessary information is stored in the dot data memory 570 using the amplitude value amp, but in another embodiment, necessary information can be stored in the dot data memory 570 using the line memory 560. That is, instead of controlling all pixels of the pixel array 520 only with the first modulation control signal MCS1 and the second modulation control signal MCS2, for example, the first modulation control signal MCS1 and the second modulation control signal MCS2 having a phase difference of 0 degrees (i.e., in phase) and a phase difference of 180 degrees from the modulated optical signal MLS can be applied to odd-numbered pixels among pixels belonging to one row, and the first modulation control signal MCS1 and the second modulation control signal MCS2 having a phase difference of 90 degrees and a phase difference of 270 degrees from the modulated optical signal MLS can be applied to even-numbered pixels. According to this control method (pseudo-4TAP), the first to fourth depth data C0 to C4 can be stored in the line memory 560, so that the position X of the dot center pixel can be determined. i , Y i and the amplitude value amp ican be stored in the dot data memory 570. In this case, since the first to fourth depth data C0 to C4 correspond to two pixels instead of one pixel, it may be disadvantageous in terms of resolution, but it has an advantage in that the line memory 560 can be used instead of the frame memory 400. On the other hand, the position X of the dot center pixel i , Y i and the amplitude value amp i The process of generating the can be substantially the same as the method described in FIG.

[0142] FIG. 20 is a flowchart showing step S250 shown in FIG. 14 in more detail.

[0143] 20, the overlapped dot center pixel remover 140 can initialize (i=1) a search point index i, which is identification information of a search point SP (S251). Here, it should be noted that each operation in FIG. 20 can be performed only for a valid search point SP_V determined in step S240, and the search point SP referred to in FIG. 20 is a valid search point SP_V.

[0144] The overlapped dot center pixel remover 140 detects the position X of the dot center pixel corresponding to the search point SP that corresponds to the current search point index. i , Y i can be obtained (S252).

[0145] The overlapped dot center pixel removal unit 140 detects the position X of the dot center pixel corresponding to the adjacent search point SP corresponding to the current search point index. j , Y j Here, the adjacent search point may refer to a search point adjacent to the search point SP corresponding to the current search point index in the upper, lower, left, right, or diagonal direction.

[0146] The overlapped dot center pixel remover 140 detects the position X of the dot center pixel corresponding to the search point SP that corresponds to the current search point index. i , Y i and the position X of the dot center pixel corresponding to the adjacent search point. j , Y j Then, the distance l between the pixels can be calculated (S253-1).

[0147] The overlapped dot center pixel remover 140 may compare the distance l between the pixels with the diameter of the dot DOT to determine whether the two pixels are close to each other (S254). Here, the diameter of the dot DOT may be a value experimentally determined in advance as the maximum diameter of the dot DOT.

[0148] If the distance l between the pixels is smaller than the dot DOT (Yes in S254), the dot center pixels overlap each other corresponding to one dot, so the overlapped dot center pixel remover 140 can remove one of the overlapped dot center pixels (S255). According to an embodiment, the overlapped dot center pixel remover 140 can select only the dot center pixel with the larger charge value from among the dot center pixels.

[0149] If the distance l between the pixels is equal to or greater than the dot DOT (No in S254), the step S255 may not be performed since the dot center pixels do not overlap each other.

[0150] The overlapping dot center pixel remover 140 can determine whether or not step S253 has been performed for all adjacent search points to the search point SP (S256).

[0151] If step S253 has not been performed for all adjacent search points for search point SP (No in S256), step S253 may be performed for other adjacent search points for search point SP. Next, steps S253 to S256 may be repeatedly performed for other adjacent search points for search point SP, and when steps S253 to S256 have been performed for all adjacent search points for search point SP, overlapped dot center pixel removal unit 140 may remove overlapped dot center pixels one by one in relation to search point SP.

[0152] If step S253 has been performed for all adjacent search points for search point SP (Yes in S256), and if step S252 has not been performed for all search points SP (No in S257), the overlapping dot center pixel remover 140 increments the current search point index i by 1 (S258) and can perform step S252 again. This operation can be repeated until step S252 has been performed for all search points. If step S252 has been performed for all search points (Yes in S257), the position X of the dot center pixel is i , Y i and the charge value B i The dot center pixel information for all the dot center pixels can be stored in the dot data memory 570 so that there is no overlap.

[0153] As in the embodiment described in Figures 13 to 20, when the charge value is used as the intensity of the depth data to determine the dot center pixel, only one frame is required instead of two frames, and the dot center pixel can be detected using only a line memory instead of a frame memory.

[0154] FIG. 21 is a block diagram of an example computing device that corresponds to the dot center pixel detector of FIG.

[0155] Referring to FIG. 21, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the dot center pixel detector 100 of FIG.

[0156] Computing device 1000 may include a processor 1010, a memory 1020, an input / output interface 1030, and a communication interface 1040.

[0157] The processor 1010 is capable of processing data and / or instructions necessary to perform the operations of the components 110-140 of the dot center pixel detector 100 of FIG.

[0158] The memory 1020 can store data and / or instructions necessary to perform operations of the components 110-140 of the dot center pixel detector 100 and can be accessed by the processor 1010. For example, the memory 1020 can be implemented with volatile memory (e.g., Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), etc.) or non-volatile memory (e.g., Programmable Read Only Memory (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), flash memory, etc.).

[0159] That is, a computer program for performing the operation of the dot center pixel detector 100 disclosed in this document can be recorded in the memory 1020 and executed and processed by the processor 1010 to realize the operation of the image processing device 100.

[0160] The input / output interface 1030 can provide an interface to connect an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010 and enable data to be transmitted and received.

[0161] The communication interface 1040 is configured to be capable of transmitting and receiving various data to and from an external device (eg, an application processor, an external memory, etc.), and may be a device capable of supporting wired or wireless communication.

[0162] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the present invention can be embodied in other specific forms without changing the technical idea or essential features of the present invention. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. a line memory that senses dot pattern light reflected from a scene and stores the generated depth data in units of line groups; a dot center pixel detector for determining a dot center pixel having the strongest depth data intensity for each of a plurality of kernels included in the line group; and a dot data memory for storing dot center pixel information including the position of the dot center pixel.

2. 2. The depth image sensing device of claim 1, wherein the intensity of the depth data is a charge value of first and second depth data generated using a first modulation control signal having a predetermined phase difference with respect to a dot pattern light irradiated on the scene and a second modulation control signal having an opposite phase to the first modulation control signal.

3. the charge value is a sum of the first depth data and the second depth data; 3. The depth image sensing device of claim 2, wherein the charge value is deemed to be a valid charge value if a difference obtained by subtracting an average of charge values ​​of pixels surrounding the pixel from the charge value is greater than a predetermined threshold difference.

4. a pixel array including a plurality of pixels for sensing the reflected dot pattern light; The depth image sensing device of claim 1 , wherein the number of lines included in the line group is smaller than the number of lines included in the pixel array.

5. The depth image sensing device of claim 1 , wherein the dot center pixel detector includes a search point manager that determines search points to be included in each of the plurality of kernels.

6. The depth image sensing device according to claim 5 , wherein a distance between the search point and an adjacent search point adjacent to the search point is smaller than a distance between dots of the dot pattern light.

7. 2. The depth image sensing device of claim 1, wherein the dot center pixel detector comprises a surrounding pixel comparator that compares intensities of the depth data of a plurality of pixels included in each of the plurality of kernels to determine the dot center pixel.

8. 2. The depth image sensing device of claim 1, wherein the dot center pixel detector includes a valid pixel determiner that compares an intensity of the depth data of the dot center pixel with a validity threshold to determine whether the dot center pixel is a valid pixel.

9. The depth image sensing device according to claim 8 , wherein the dot center pixel detector includes a duplicate dot center pixel remover that removes duplicate dot center pixels from the dot center pixels that are the effective pixels.

10. The depth image sensing device according to claim 9 , wherein the overlapping dot center pixels are pixels whose distance from other dot center pixels is smaller than a diameter of a dot of the dot pattern light.

11. a dot center pixel detector for determining a dot center pixel having the strongest depth data intensity for each of a plurality of kernels; a depth image generator that uses the depth data of the dot center pixel to calculate distance to a scene.

12. 12. The image signal processor of claim 11, wherein the intensity of the depth data is an amplitude value of the first to fourth depth data generated using first to fourth modulation control signals having different phase differences from a dot pattern light irradiated on the scene.

13. 12. The image signal processor of claim 11, wherein the dot center pixel detector includes a search point manager that determines search points to be included in each of the plurality of kernels.

14. 14. The image signal processor of claim 13, wherein a distance between the search point and an adjacent search point adjacent to the search point is smaller than a distance between dots of a dot pattern light irradiated onto the scene.

15. 12. The image signal processor of claim 11, wherein the dot center pixel detector comprises a surrounding pixel comparator that compares intensities of the depth data of a plurality of pixels included in each of the plurality of kernels to determine the dot center pixel.

16. 12. The image signal processor of claim 11, wherein the dot center pixel detector includes a valid pixel determiner that compares an intensity of the depth data of the dot center pixel with a validity threshold to determine whether the dot center pixel is a valid pixel.

17. 17. The image signal processor of claim 16, wherein the dot center pixel detector includes a duplicate dot center pixel remover that removes duplicate dot center pixels from the dot center pixels that are the effective pixels.

18. 18. The image signal processor of claim 17, wherein the overlapping dot center pixels are pixels that are in the same position as other dot center pixels.

19. determining a dot center pixel having the strongest depth data intensity for each of a plurality of kernels; comparing the intensity of the depth data of the dot center pixel with a significance threshold to determine whether the dot center pixel is a significant pixel; removing overlapping dot center pixels from among the dot center pixels that are effective pixels; and calculating a distance to a scene using the depth data of the dot center pixel.

20. 20. The image signal processing method of claim 19, wherein the effective pixel is determined by repeating a step of determining a dot center pixel having the strongest intensity of the depth data and a step of determining whether the dot center pixel is a valid pixel a plurality of times.

Images