Depth image sensing device, image signal processor, and image signal processing method

Abstract

The invention relates to a depth image sensing device, an image signal processor and an image signal processing method. A depth image sensing apparatus includes: a row memory configured to store depth data generated by detecting dot pattern light reflected from a scene in units of row groups; a point center pixel detector configured to determine a point center pixel having depth data of the highest intensity in each of the plurality of kernels included in the row group; and a point data memory configured to store point center pixel information including a position of the point center pixel.

Description

Technical Field

[0001] Technologies and embodiments disclosed in this patent document generally relate to a depth image sensing device, an image signal processor, and an image signal processing method capable of processing information about a scene and information about a distance to the scene. Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer and communication industries, the demand for high-performance image sensing devices is increasing in various devices such as smartphones, digital cameras, game consoles, IOT (Internet of Things), robots, security cameras and medical micro cameras.

[0003] The image sensing device can be used to acquire a color image of a scene or a depth image of the scene. An image sensing device capable of acquiring a depth image may use a separate light source and may consume a large amount of power to operate the separate light source.

[0004] In particular, when an image sensing device is mounted on a mobile device, reducing power consumption required to operate a light source has become an important issue. Summary of the invention

[0005] According to an embodiment of the disclosed technology, a depth image sensing device may include: a row memory configured to store depth data generated by detecting dot pattern light reflected from a scene in units of row groups; a dot center pixel detector configured to determine a dot center pixel having the highest intensity depth data in each of a plurality of kernels included in the row group; and a dot data memory configured to store dot center pixel information including the position of the dot center pixel.

[0006] According to another embodiment of the disclosed technology, an image signal processor may include: a point center pixel detector configured to determine a point center pixel having depth data of the highest intensity in each of a plurality of kernels; and a depth image generator configured to calculate a distance to a scene using the depth data of the point center pixel.

[0007] According to another embodiment of the disclosed technology, an image signal processing method may include the following steps: determining at least one point center pixel having the highest intensity of depth data in each of a plurality of kernels; determining whether the point center pixel is a valid pixel by comparing the intensity of the depth data of the point center pixel with a valid threshold; removing overlapping point center pixels from the point center pixels that are valid pixels; and calculating the distance to the scene using the depth data of the point center pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features and advantageous aspects of the disclosed technology will become readily apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0009] Figure 1 is a schematic diagram illustrating an example of an imaging device based on some embodiments of the disclosed technology.

[0010] Figure 2 The present invention is to show some implementations based on the disclosed technology. Figure 1 A timing diagram of an example of depth data generated when an imaging device uses a direct time-of-flight (TOF) method.

[0011] Figure 3 The present invention is to show some implementations based on the disclosed technology. Figure 1 A timing diagram of an example of depth data generated when an imaging device uses an indirect time-of-flight (TOF) method.

[0012] Figure 4 is a block diagram illustrating an example of a point center pixel detector based on some implementations of the disclosed technology.

[0013] Figure 5 The following are some examples of the operation of some implementations based on the disclosed technology. Figure 4 A flow chart of an example of a method of a point center pixel detector is shown.

[0014] Figure 6 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S20 is shown.

[0015] Figure 7 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S30 is shown.

[0016] Figure 8 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S50 is shown in FIG.

[0017] Fig. 9 Some implementations based on the disclosed technology include Figure 4 A block diagram of an example of an imaging device with a dot-centered pixel detector is shown.

[0018] Fig.10 The following are some examples of the operation of some implementations based on the disclosed technology. Fig. 9 A flow chart of an example of a method of an imaging apparatus is shown.

[0019] Fig.11 Some implementations based on the disclosed technology are shown in Fig.10 A flowchart of an example of operation S130 is shown.

[0020] Fig.12 Some implementations based on the disclosed technology are shown in Fig.10 A flow chart of an example of operation S150 is shown.

[0021] Fig.13 Some implementations based on the disclosed technology include Figure 4 A block diagram of another example of an imaging device with a dot-center pixel detector is shown.

[0022] Fig.14 The following are some examples of the operation of some implementations based on the disclosed technology. Fig.13 A flow chart of an example of a method of an imaging apparatus is shown.

[0023] Fig.15 is a conceptual diagram illustrating an example of operations performed on a first row group based on some implementations of the disclosed technology.

[0024] Fig.16 is a conceptual diagram illustrating an example of operations performed on a second row group based on some implementations of the disclosed technology.

[0025] Fig.17 and Fig.18 Some implementations based on the disclosed technology are shown in Fig.14 A conceptual diagram of an example of operation S250 is shown.

[0026] Fig.19 Some implementations based on the disclosed technology are shown in Fig.14 A flowchart of an example of operation S230 is shown.

[0027] Fig. 20 Some implementations based on the disclosed technology are shown in Fig.14 A flow chart of an example of operation S250 is shown.

[0028] Fig.21 It is shown that Figure 4 A block diagram of an example of a computational device for a point center pixel detector. DETAILED DESCRIPTION

[0029] This patent document provides embodiments and examples of a depth image sensing device, an image signal processor, and an image signal processing method that can process information about a scene and information about the distance to the scene, which can be used in a configuration that substantially solves one or more technical or engineering problems and alleviates limitations or disadvantages encountered in some other image sensing devices. Some embodiments of the disclosed technology relate to a depth image sensing device, an image signal processor, and an image signal processing method that can improve depth sensing performance while reducing the power consumption required to operate a light source. The disclosed technology provides various embodiments of a depth image sensing device, an image signal processor, and an image signal processing method that can efficiently and accurately detect distance regardless of the scene.

[0030] Reference will now be made in detail to embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are illustrated as examples in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0031] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0032] Various embodiments of the disclosed technology relate to a depth image sensing device, an image signal processor, and an image signal processing method capable of improving depth sensing performance while reducing power consumption required to operate a light source.

[0033] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are exemplary and explanatory and are intended to provide further explanation of the disclosure as claimed.

[0034] Figure 1 is a block diagram illustrating an example of an imaging device 1 based on some embodiments of the disclosed technology.

[0035] Reference Figure 1 The imaging device 1 may be a device for acquiring a depth image of a scene, and the depth image may be an image representing a sense of distance to the scene. That is, the user may visually check the distance to the scene through the depth image.

[0036] In order to acquire a depth image, the imaging device 1 may include a light source module 2 and a camera 3 .

[0037] The light source module 2 may emit light (MLS) toward a scene, and the camera 3 may generate an electrical signal by detecting the light (MLS_R) after the emitted light is reflected from the scene. A processor (not shown) disposed inside or outside the camera 3 may generate a depth image based on the electrical signal generated by the camera 3.

[0038] The light source module 2 may consume a large amount of power to emit light. In order to reduce such power consumption, the light source module 2 may emit dot pattern light to the scene. The dot pattern light may refer to a plurality of lights that are discontinuously irradiated in a specific area, rather than a single surface light that is continuously irradiated in a specific area. As a result, the dot pattern light may appear as light beams, each of which has a dot shape in the scene and the camera 3.

[0039] In order to accurately sense the dot pattern light, it is necessary to accurately detect the center of each dot constituting the dot pattern light. However, since each dot of the dot pattern light incident on the camera 3 has scene-dependent characteristics, it may be difficult to accurately detect the center of each dot using a fixed method. The scene-dependent characteristics may occur due to the position difference between the light source module 2 and the camera 3 (i.e., the deviation of the dot position), or may also occur due to the distance difference between the camera 3 and the scene (i.e., the deviation of the dot size). The disclosed technology shows a method that can accurately sense the dot pattern light regardless of the scene-dependent characteristics of each dot.

[0040] Figure 2 The present invention is to show some implementations based on the disclosed technology. Figure 1 A timing diagram of an example of depth data generated when an imaging device uses a direct time-of-flight (TOF) method.

[0041] Reference Figure 2 , the imaging device 1 may generate depth data serving as a basis for a depth image by using a direct TOF method, and the direct TOF method may be a method for directly measuring a time difference between a time point at which a modulated light signal (MLS) is emitted to a scene and a time point at which the modulated light signal (MLS) is incident on the camera 3 after being reflected from the scene. In this case, the depth data may be pixel data obtained by detecting the modulated light signal (MLS) reflected from the scene.

[0042] The camera 3 may include a plurality of direct pixels for a direct TOF method. In some implementations, each of the plurality of direct pixels may be a pixel including a single photon avalanche diode (SPAD). When the electric field is increased by applying a reverse bias voltage to the SPAD, impact ionization may occur, in which a pair of electrons and holes (hereinafter referred to as electron-hole pairs) are generated while electrons generated by photons due to the strong electric field move. Specifically, in a SPAD operating in a Geiger mode with a reverse bias voltage applied that is higher than the breakdown voltage, electrons and holes generated by impact ionization caused by collisions with carriers (ie, electrons or holes) generated by incident light may collide with each other, and countless carriers may be generated. As a result, even if a single photon is incident on the SPAD, the single photon may trigger avalanche breakdown, thereby generating a measurable current pulse.

[0043] exist Figure 2 In the embodiment, the modulated light signal (MLS) emitted from the light source module 2 may have a pulse shape with a relatively large amplitude and a small pulse width, and a time point at which a pulse is generated may be defined as a reference pulse time (RPT).

[0044] The light source module 2 may emit a modulated light signal (MLS) to the scene, and the camera 3 may detect light reflected from the scene (i.e., reflected light), and may generate pixel data (PD) corresponding to the generated current pulse. In addition, the camera 3 may determine a time point at which a value of the pixel data PD is equal to or greater than a threshold data value as a pulse sensing time (PST), calculate a time of flight (TOF) representing a time difference between a reference pulse time (RPT) and the pulse sensing time (PST), and calculate the calculated TOF and the speed of light (luminous flux) (for example, a value obtained by dividing the TOF by "2" may be multiplied by the speed of light) to calculate the distance between the scene and the camera 3.

[0045] Figure 3 The present invention is to show some implementations based on the disclosed technology. Figure 1 A timing diagram of an example of depth data generated when an imaging device uses an indirect time-of-flight (TOF) method.

[0046] Reference Figure 3 , the imaging device 1 may use an indirect TOF method to generate depth data used as a basis for a depth image. The indirect TOF method may be a method of calculating a phase difference between a modulated light signal (MLS) emitted to a scene and a reflected modulated light signal (MLS_R) incident on the camera 3 after the modulated light signal (MLS) is reflected from the scene. In this case, the depth data may be pixel data corresponding to photocharges captured by pixels controlled by a modulation control signal having a constant phase difference (e.g., 0 degrees, 90 degrees, 180 degrees, or 270 degrees) relative to the modulated light signal (MLS).

[0047] The camera 3 may include a plurality of indirect pixels for an indirect TOF scheme. In some implementations, each of the plurality of indirect pixels may be a pixel having a 2-tap structure. The pixel having a 2-tap structure may include a photoelectric conversion element that converts incident light into photocharges, two transistors that are turned on by a first modulation control signal MCS1 and a second modulation control signal MCS2 having opposite phases to transmit photocharges at different time points, and two output circuits that convert the photocharges output from the two transistors into electrical signals and output the resulting electrical signals. For example, the first modulation control signal MCS1 may have a 0 degree phase difference (i.e., in phase) relative to the modulated light signal (MLS), and the second modulation control signal MCS2 may have a 180 degree phase difference relative to the modulated light signal (MLS). Alternatively, the first modulation control signal MCS1 may have a 90 degree phase difference relative to the modulated light signal (MLS), and the second modulation control signal MCS2 may have a 270 degree phase difference relative to the modulated light signal (MLS).

[0048] Figure 3 The modulated light signal (MLS) shown may have Figure 2 The modulated light signal (MLS) shown has a pulse with a smaller amplitude and a larger pulse width than the modulated light signal (MLS). Due to the time of flight (TOF) obtained when the modulated light signal (MLS) is incident on the camera 3 after being reflected from the scene, the reflected modulated light signal (MLS_R) may have a predetermined phase difference (θ) with the modulated light signal (MLS). The operation period in which the camera 3 calculates the distance to the scene may be divided into a first sensing period SP1 and a second sensing period SP2. In addition, it is assumed that the phase difference (θ) between the modulated light signal (MLS) and the reflected modulated light signal (MLS_R) is maintained constant during the first sensing period SP1 and the second sensing period SP2.

[0049] In the first sensing period SP1, the first modulation control signal (MCS1) may have the same phase as the modulated light signal (MLS) (i.e., a phase difference of 0 degrees). That is, the first modulation control signal (MCS1) may be in phase with the modulated light signal (MLS). Then, in the first period PR1, the first modulation control signal (MCS1) may have an enable voltage (or a logic high level). The first depth data (C0) indicating the amount of photocharge generated in the first period PR1 may be generated by the first modulation control signal (MCS1) having the enable voltage.

[0050] In the first sensing period SP1, the second modulation control signal (MCS2) may have a phase opposite to that of the modulated light signal (MLS) (i.e., a 180-degree phase difference). Then, in the second period PR2, the second modulation control signal (MCS2) may have an enable voltage (or a logic high level). The third depth data (C2) indicating the amount of photocharge generated in the second period PR2 may be generated by the second modulation control signal (MCS2) having the enable voltage.

[0051] In the second sensing period SP2, the first modulation control signal (MCS1) may have a phase difference (i.e., a 90-degree phase difference) with respect to the modulated light signal (MLS). Then, in the third period PR3, the first modulation control signal (MCS1) may have an enable voltage (or a logic high level). The second depth data (C1) indicating the amount of photocharge generated in the third period PR3 may be generated by the first modulation control signal (MCS1) having the enable voltage.

[0052] In the second sensing period SP2, the second modulation control signal (MCS2) may have a phase difference (i.e., a 270-degree phase difference) relative to the modulated light signal (MLS). Then, in the fourth period PR4, the second modulation control signal (MCS2) may have an enable voltage (or a logic high level). The fourth depth data (C3) indicating the amount of photocharge generated in the fourth period PR4 may be generated by the second modulation control signal (MCS2) having the enable voltage.

[0053] That is, the reflected modulated optical signal (MLS_R) may be captured by dividing the first sensing period SP1 into the first period PR1 and the second period PR2 , and may be captured by dividing the second sensing period SP2 into the third period PR3 and the fourth period PR4 .

[0054] The phase difference (θ) between the modulated light signal (MLS) from the first to fourth depth data (C0-C3) and the reflected modulated light signal (MLS_R) may be calculated by the following Equation 1.

[0055] [Formula 1]]

[0056]

[0057] In addition, the distance (d) between the camera 3 and the scene can be calculated by the following equation 2 using the phase difference (θ).

[0058] [Formula 2]

[0059]

[0060] In Formula 2, "c" may be the speed of light (luminous flux), and "fm" may be the frequency of the modulated optical signal (MLS).

[0061] In addition, the amplitude value (amp) of the reflected modulated light signal (MLS_R) can be calculated by the following formula 3.

[0062] [Formula 3]

[0063]

[0064] In addition, the charge value (B) representing the total amount of photocharges generated by reflecting the modulated light signal (MLS_R) may correspond to the sum of the first and third depth data (C0+C2) or the sum of the second and fourth depth data (C1+C3).

[0065] According to Figure 3 The phase difference detection method (4-phase modulation method) shown can use differential values ​​such as (C1-C3) and / or (C0-C2) to calculate the phase difference, so that the component caused by the background noise included in each of the first to fourth depth data (C0~C3) can be removed, thereby calculating the distance more accurately.

[0066] Figure 4 is a block diagram illustrating an example of a point center pixel detector based on some implementations of the disclosed technology. Figure 5 The following are some examples of the operation of some implementations based on the disclosed technology. Figure 4 A flow chart of an example of a method of a point center pixel detector is shown. Figure 6 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S20 is shown. Figure 7 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S30 is shown. Figure 8 Some implementations based on the disclosed technology are shown in Figure 5 A conceptual diagram of an example of operation S50 is shown in FIG.

[0067] Reference Figure 4 and Figure 5 , although the point center pixel detector 100 may be included in Figure 1 However, other implementations are also possible. It should be noted that the point center pixel detector 100 may be located outside the camera 3.

[0068] The point center pixel detector 100 may receive and analyze the depth data to determine the point center pixel in the depth data, and may generate and output the point center pixel information. The camera 3 may include a pixel array in which a plurality of pixels are arranged in a matrix shape including a plurality of rows and a plurality of columns to obtain the depth data of the scene. Each of the plurality of pixels included in the pixel array may generate the depth data of a scene, and the depth data may constitute a frame. The depth data to be input to the point center pixel detector 100 may correspond to a frame, and the depth data corresponding to a frame may be sequentially input to the point center pixel detector 100 through a serial interface without being input simultaneously.

[0069] Since the reflected modulated light signal input to the camera 3 corresponds to the dot pattern light, there may be a region where light is incident in one frame (eg, Figure 6 DOT in the image) and areas without incident light (e.g. Figure 6 The remaining area outside the DOT). Even within the area where light is incident, the intensity of the depth data at the dot center pixel located at the center of the light incident area may have the highest value, and the intensity of the depth data may gradually decrease toward the periphery of the dot center pixel. As the intensity of the depth data increases, the signal-to-noise ratio (SNR) may be improved so that the distance to the scene calculated using the depth data of the dot center pixel may be most effective. Therefore, in the process of detecting the distance using the dot pattern light, it may be considered that the operation of determining the dot center pixel by the dot center pixel detector 100 is important.

[0070] The point center pixel information may include position data of the point center pixel and depth data of the point center pixel. In some implementations, the point center pixel information may also include data obtained by processing the depth data of the point center pixel.

[0071] The point center pixel detector 100 may include a search point management unit 110, a nearby pixel comparison unit 120, a valid pixel determination unit 130, and an overlapped point center pixel removal unit 140. The point center pixel detector 100 may be implemented in hardware, software, and / or a combination thereof.

[0072] The search point management unit 110 may acquire depth data corresponding to one frame (S10). Here, the following description will be given on the premise that the search point management unit 110 acquires depth data in units of one frame, but the scope of the disclosed technology is not limited thereto. Alternatively, the search point management unit 110 may also acquire depth data based on a unit smaller than one frame (i.e., depth data corresponding to the storage capacity of the line memory).

[0073] Figure 6The depth data corresponding to one frame is shown, and each depth data corresponding to the dot pattern light may be represented by a dot (DOT). The brightness within the dot (DOT) may represent the intensity of the depth data, and as the brightness decreases (ie, becomes darker), the intensity of the depth data may become stronger. Figure 6 In the present invention, for the convenience of description, the number of points (DOT) and the positions of the points (DOT), the number of search points (SP) and the positions of the search points (SP) are merely examples, and the scope of the disclosed technology is not limited thereto.

[0074] When each pixel is a direct pixel, the intensity of the depth data may be pixel data obtained by detecting a modulated light signal (MLS) reflected from a scene. In some other implementations, when each pixel is an indirect pixel, the intensity of the depth data may be an amplitude value or charge value calculated based on pixel data (i.e., first depth data C0 to fourth depth data C3) corresponding to photocharges captured by pixels controlled by a modulation control signal having a constant phase difference (e.g., 0 degrees, 90 degrees, 180 degrees, or 270 degrees) relative to the modulated light signal (MLS).

[0075] The search point management unit 110 may determine or manage the initial positions of the search points arranged according to a predetermined rule with respect to the depth data corresponding to one frame (S20). Here, each search point may refer to the position of a pixel used as a unit of a center pixel of the search point. In some other implementations, when the search point is fixed in advance, operation S20 may be omitted.

[0076] exist Figure 6 In the embodiment, the search points (SP) may be arranged to correspond to the arrangement of the dots (DOT) constituting the dot pattern light. The search points (SP) may be respectively arranged at the intersections of the grid pattern to correspond to the points arranged in a line in each of the horizontal and vertical directions. In some implementations, the interval between adjacent search points (SP) in the horizontal direction may be set to be smaller than the interval between adjacent dots (DOT) arranged in the horizontal direction, and the interval between adjacent search points (SP) in the vertical direction may be set to be smaller than the interval between adjacent dots (DOT) arranged in the vertical direction. This is to prevent the existence of dot center pixels (i.e., missed pixels) that are not searched by any search point. Here, the interval between the dots (DOT) may be experimentally determined to be the minimum interval between the dots (DOT). The arrangement format of the search points (SP) is not limited to Figure 6 , which may be arranged, for example, in a zigzag pattern.

[0077] The search point management unit 110 may determine the number of search points (SP) and the positions of the search points (SP) in a predetermined manner according to the arrangement format of the points constituting the dot pattern light, and may store the determined number of search points (SP) and / or the determined positions of the search points (SP). In some other implementations, the search point management unit 110 may dynamically adjust the positions of the pre-stored search points (SP) and / or the number of pre-stored search points (SP). For example, when the number of dot center pixels included in the dot center pixel information is less than or equal to a predetermined ratio (e.g., 50%) compared to the total number of dots (DOT), the search point management unit 110 may increase the number of search points (SP) and may reduce the size of the interval between adjacent search points (SP).

[0078] The nearby pixel comparison unit 120 may determine the point center pixel (DCP) of each search point (SP) based on the depth data of the pixels (i.e., the neighboring pixels) located within a predetermined range of the corresponding search point (SP). In some implementations, the nearby pixel comparison unit 120 may calculate the intensity of the depth data of the neighboring pixels, and may determine the point center pixel (DCP) by comparing the intensities of the depth data of the neighboring pixels with each other. That is, the nearby pixel comparison unit 120 may determine one pixel having the maximum intensity of the depth data among the pixels surrounding the search point (SP) as the point center pixel DCP (S30).

[0079] exist Figure 6 In the embodiment, a set (or aggregation) of pixels within a specific range of each search point (SP) may be defined as a kernel (KN). In some implementations, the horizontal length of the kernel (KN) may be set to be equal to or greater than the interval between horizontally adjacent search points (SP) and less than twice the interval, and the vertical length of the kernel (KN) may be set to be equal to or greater than the interval between vertically adjacent search points (SP) and less than twice the interval. This is only an example, and the size of the kernel (KN) may be determined experimentally to minimize the number of unnecessary calculations (e.g., repeated detection of dot center pixels DCP) and the number of undetected dot center pixels (DCP). In an embodiment, there may be multiple kernels (KN).

[0080] The valid pixel determination unit 130 may compare the intensity of the depth data of each dot center pixel (DCP) determined by the nearby pixel comparison unit 120 with the valid threshold value, and may determine whether each dot center pixel (DCP) is a valid pixel (S40). The valid threshold value may be a predetermined fixed value considering the maximum value and the minimum value of the intensity of the depth data, the signal-to-noise ratio (SNR), etc. In another embodiment, when the number of dot center pixels (DCP) included in the dot center pixel information is less than or equal to a predetermined ratio (e.g., 50%) compared to the total number of dots (DOT), the valid pixel determination unit 130 may reduce the valid threshold value.

[0081] In another embodiment, when the number of dot center pixels (DCP) is less than or equal to a predetermined ratio (e.g., 50%) compared to the total number of dots (DOT), the nearby pixel comparison unit 120 may expand the predetermined range of the corresponding search point (SP) to determine the dot center pixels (DCP) of each search point (SP), and may repeatedly perform operations S30 and S40. Therefore, the dot center pixel (DCP) may be detected more accurately while minimizing the resources consumed to determine the dot center pixel (DCP).

[0082] exist Figure 6 and Figure 7 , arrows are depicted connecting the determined dot center pixels (DCP) to the corresponding search points (SP) in the kernel (KN) including the corresponding search points (SP).

[0083] If the intensity of the depth data of the point center pixel (DCP) of the search point (SP) is greater than or equal to the valid threshold, the valid pixel determination unit 130 may determine the corresponding search point as a valid search point (SP_V). In addition, the point center pixel (DCP) corresponding to the valid search point (SP_V) may be determined as a valid pixel.

[0084] If the intensity of the depth data of the point center pixel (DCP) of the search point (SP) is less than the valid threshold, the valid pixel determination unit 130 may determine the corresponding search point as an invalid search point (SP_I). In addition, the point center pixel (DCP) corresponding to the invalid search point (SP_I) may be determined as an invalid pixel.

[0085] The overlapping point center pixel removal unit 140 may remove the overlapping point center pixel from among dot center pixels (DCP) used as valid pixels (S50).

[0086] exist Figure 8 In the overlap point center pixel DCP_O (in Figure 8The dot center pixel (indicated by double circles in the figure) may refer to pixels corresponding to multiple valid search points (SP_V). The normal dot center pixel (DCP_N) may refer to pixels corresponding to one valid search point (SP_V). When the dot center pixel information includes information about a dot center pixel (DCP) that is redundant and substantially the same as the overlapping dot center pixel (DCP_O), the distance to the scene may be calculated multiple times unnecessarily. Therefore, the overlapping dot center pixel removal unit 140 may select only one dot center pixel from among the overlapping dot center pixels (DCP_O).

[0087] Here, when the positions of the dot center pixels (DCP) are the same or are within a predetermined distance (e.g., the diameter of each dot of the dot pattern light), the overlapping point center pixel removal unit 140 may determine the corresponding dot center pixel (DCP) as the overlapping point center pixel (DCP_O). In some implementations, when the positions of the dot center pixels (DCP) are the same, the overlapping point center pixel removal unit 140 may select only the dot center pixel (DCP) corresponding to the search point (SP) having the minimum index or the maximum index (i.e., the minimum identification (ID) number or the maximum ID number) of the search point (SP). In some other implementations, when the dot center pixel (DCP) is within a predetermined distance, the overlapping point center pixel removal unit 140 may select only the dot center pixel (DCP) having the maximum intensity of the depth data.

[0088] Based on some implementations of the disclosed technology, the point center pixel detector 100 can correctly calculate the point center pixel and the distance to the scene regardless of scene-related characteristics or other distortions (for example, lens distortion, changes in the relative position relationship between the light source module 2 and the camera 3, etc.), thereby minimizing the number of unnecessary calculations.

[0089] Fig. 9 Some implementations based on the disclosed technology include Figure 4 A block diagram of an example of an imaging device with a dot-center pixel detector 100 is shown.

[0090] Reference Fig. 9 , the imaging device 900 may include a depth image sensing device 200 , an image signal processor 300 , and a frame memory 400 .

[0091] The depth image sensing device 200 may generate depth data of a target object (OBJ) by using at least one depth sensing technology. The depth image sensing device 200 may include a light source module 210 , a pixel array 220 , a pixel driver 230 , a timing controller 240 , and a pixel readout circuit 250 .

[0092] The light source module 210 may emit light to the target object (OBJ) upon receiving a control signal from the timing controller 240. The light source module 210 may be a laser diode (LD) or a light emitting diode (LED) for emitting light having a specific wavelength band (e.g., near infrared (NIR) light, infrared (IR) light, or visible light), or may be a near infrared laser (NIR), a point light source, a monochromatic light source or a monochromator combined with a white light, and a combination of other laser sources. For example, the light source module 210 may emit infrared light having a wavelength of 800nm ​​to 1000nm. In addition, the modulated light signal (MLS) may be a pulsed light having a predetermined modulation characteristic (e.g., a predetermined waveform, wavelength, period, amplitude, frequency, phase, duty cycle, etc.).

[0093] The pixel array 220 may include a plurality of pixels that are sequentially arranged in a two-dimensional (2D) matrix structure (eg, sequentially arranged in a row direction and a column direction). Fig. 9 In the following embodiments of the drawings, each pixel may be a direct pixel or an indirect pixel. Although other implementations are also possible, for the convenience of description, the following description assumes that each pixel is an indirect pixel.

[0094] Each pixel of the pixel array 220 can perform photoelectric conversion on a reflected modulated light signal (MLS_R) received through a lens module (not shown) when receiving a control signal from the pixel driver 230, generate a pixel signal serving as an electrical signal corresponding to the intensity of the reflected modulated light signal (MLS_R), and output the pixel signal to the pixel readout circuit 250.

[0095] The pixel driver 230 may generate a signal for controlling each pixel of the pixel array 220 under the control of the timing controller 240, and may supply the generated signal to the pixel array 220. In particular, the pixel driver 230 may generate a first modulation control signal (MCS1) and a second modulation control signal (MCS2) for controlling the timing at which each tap captures photocharges, and may supply the first modulation control signal (MCS1) and the second modulation control signal (MCS2) to the pixel array 220.

[0096] The timing controller 240 may control the overall operation of the depth image sensing device 200 by controlling the light source module 210 , the pixel driver 230 , and the pixel readout circuit 250 .

[0097] The pixel readout circuit 250 may process analog pixel signals output from each pixel to generate depth data corresponding to the pixel signals. For example, the pixel readout circuit 250 may include an analog-to-digital (AD) converter for AD-converting the pixel signals into depth data.

[0098] The image signal processor 300 may collect the depth data received from the depth image sensing device 200 to generate a depth image representing the distance to the target object (OBJ). In addition, the image signal processor 300 may perform image signal processing for noise removal and image quality improvement on the generated depth image.

[0099] The depth image output from the image signal processor 300 may be stored in an internal or external memory of the imaging device 900 or a device equipped with the imaging device 900 according to a user's request or by automation, or may be displayed on a display according to a user's request or by automation. Alternatively, the depth image output from the image signal processor 300 may be used to control the operation of the imaging device 900 or the operation of a device equipped with the imaging device 900.

[0100] The image signal processor 300 may include the dot center pixel detector 100. The image signal processor 300 may store the depth data received from the depth image sensing device 200 in the frame memory 400, and then may provide the depth data corresponding to one frame to the dot center pixel detector 100. The dot center pixel detector 100 may generate dot center pixel information as information about a plurality of dot center pixels for the depth data corresponding to one frame.

[0101] The depth image generator 310 may calculate the phase difference between the modulated light signal (MLS) and the reflected modulated light signal (MLS_R) of each of a plurality of point center pixels with reference to the point center pixel information, calculate the distance between the target object (OBJ) and the depth image sensing device 200 based on the calculated phase difference, and thereby generate a depth image.

[0102] The frame memory 400 may receive and store the depth data generated by the depth image sensing device 200 through the image signal processor 300. The frame memory 400 may have a storage capacity capable of storing depth data corresponding to one frame. In addition, the frame memory 400 may store not only data required for the operation of the image signal processor 300 (e.g., search point information, point center pixel information, etc.), but also instructions.

[0103] The imaging device 900 may correspond to Figure 1 This is a specific implementation example of the imaging device 1 described in , and may be an embodiment related to a case where the dot center pixel detector 100 is included in the image signal processor 300 .

[0104] Fig.10 The following are some examples of the operation of some implementations based on the disclosed technology. Fig. 9 A flow chart of an example of a method of an imaging apparatus is shown.

[0105] Reference Fig.10, except for some operations, Fig.10 The method of operating the imaging device is shown in Figure 5 The method of operating the point center pixel detector described in is basically the same. Therefore, its redundant description will be omitted in this article, and the following description will focus on Figure 5 Those different characteristic operations.

[0106] First, the depth image sensing device 200 may generate first depth data (C0) and third depth data (C2) in the first sensing period SP1, and the image signal processor 300 may generate the first depth data (C0) and third depth data (C2) in the frame memory 400 (S110).

[0107] The depth image sensing device 200 may generate second depth data (C1) and fourth depth data (C3) in the second sensing period SP2, and the image signal processor 300 may store the second depth data (C1) and fourth depth data (C3) in the frame memory 400 (S112).

[0108] The search point management unit 110 may determine or manage an initial position of a search point according to a predetermined rule with respect to first to fourth depth data C0 to C3 corresponding to one frame ( S120 ).

[0109] The nearby pixel comparison unit 120 may determine a dot center pixel (DCP) of each search point SP based on depth data of pixels located within a predetermined range of the corresponding search point SP ( S130 ).

[0110] The valid pixel determination unit 130 may compare the intensity of the depth data of each dot center pixel (DCP) determined by the nearby pixel comparison unit 120 with a valid threshold value to determine whether each dot center pixel (DCP) is a valid pixel (S140).

[0111] In some implementations, when the number of dot center pixels (DCP) is less than or equal to a predetermined ratio (e.g., 50%) compared to the total number of dots (DOT), the nearby pixel comparison unit 120 may expand the predetermined range of the corresponding search point (SP) for determining the dot center pixel (DCP) of each search point (SP), and may repeatedly perform operations S130 and S140. Therefore, an imaging device based on some implementations of the disclosed technology may detect the dot center pixel (DCP) more accurately while minimizing the resources consumed by determining the dot center pixel (DCP).

[0112] The overlapping point center pixel removal unit 140 may remove the overlapping point center pixel from among dot center pixels (DCP) used as valid pixels (S150).

[0113] Fig.11 Some implementations based on the disclosed technology are shown in Fig.10 A flowchart of an example of operation S130 is shown.

[0114] Reference Fig.11 , the nearby pixel comparing unit 120 may initialize (ie, i=1) a search point index (i) serving as identification (ID) information of the search point SP (S131). Figure 6 The multiple search points (SP) shown may have different search point indices. Figure 6 It can be seen that the search point index of the leftmost search point SP of the first row may be set to "1" and may be sequentially increased by 1 as moving toward the right end of the first row. The search point index of the leftmost search point SP of the second row may be set to "12" and may be sequentially increased by 1 as moving toward the right end of the second row. In this way, it is assumed that the search point indexes of a total of 77 search points (SP) are given.

[0115] The nearby pixel comparison unit 120 can obtain the position (X) of the search point SP corresponding to the current search point index. i ,Y i ), and the amplitude value (amp) of the search point SP can be calculated i )(S132). Amplitude value (amp i ) can be calculated by using the first to fourth depth data (C0-C3) of the search point SP through Equation 3. In addition, the nearby pixel comparison unit 120 can calculate the position (X i ,Y i ) is updated to the position of the center pixel of the point, and the amplitude value (amp i ) is updated to the amplitude value of the point center pixel.

[0116] The nearby pixel comparison unit 120 may obtain any pixel (X) within the kernel (KN) including the search point SP. i+Δx ,Y i+Δy )'s first to fourth depth data (C0~C3) (S133).

[0117] The neighboring pixel comparison unit 120 may use the pixel (X i+Δx ,Y i+Δy ) to determine the pixel (X i+Δx ,Y i+Δy )’s amplitude value (amp_cal) (S134).

[0118] When the calculated amplitude value (amp_cal) is greater than the amplitude value of the center pixel (amp i )(S135 is yes), the neighboring pixel comparison unit 120 may respectively i+Δx,Y i+Δy ) is updated to the position of the center pixel of the point (X i ,Y i ), and the calculated pixel (X i+Δx ,Y i+Δy ) is updated to the amplitude value (amp_cal) of the pixel at the center of the point i )(S136).

[0119] When the calculated amplitude value (amp_cal) is less than or equal to the amplitude value of the center pixel (amp i )(S135 is No), the position of the center pixel of the point can be maintained (X i ,Y i ) and amplitude value (amp i ) without performing operation S136.

[0120] The nearby pixel comparison unit 120 may determine whether operation S133 has been performed on all pixels included in the kernel (KN) having the search point SP ( S137 ).

[0121] If operation S133 is not performed on all pixels in the kernel KN (S137: No), the kernel KN may be operated on except for the pixels (X) on which operation S133 has been performed. i+Δx ,Y i+Δy ) other pixels except for the kernel KN. Then, operations S133 to S137 may be repeatedly performed on all pixels in the kernel KN, and when operations S133 to S137 are performed on all pixels in the kernel KN, the kernel KN having the highest amplitude value (amp i ) can be determined as the point center pixel, and the position of the corresponding point center pixel (X i ,Y i ) and amplitude value (amp i ) may be stored in the frame memory 400. In an embodiment, a frame having the highest amplitude value (amp) may be determined in the plurality of kernels (KN) (eg, in each of the plurality of kernels (KN)). i ) pixels.

[0122] When operation S133 has been performed on all pixels in the kernel KN (S137 is yes) and operation S132 has not been performed on all search points (S138 is no), the nearby pixel comparison unit 120 may increase the current search point index (i) by 1 (S139), and the operation process may return to operation S132. The above operation may be repeated until operation S132 is performed on all search points, and if operation S132 is performed on all search points (S138 is yes), the position (X) of the point center pixel of each of all search points is calculated. i ,Yi ) and amplitude value (amp i ) can be stored in the frame memory 400.

[0123] Fig.12 Some implementations based on the disclosed technology are shown in Fig.10 A flow chart of an example of operation S150 is shown.

[0124] Reference Fig.12 , the overlapping point center pixel removal unit 140 may initialize the search point index (i) used as identification (ID) information of the search point SP (ie, i=1) (S151). Here, it should be noted that the operation may be performed only for the valid search point (SP_V) determined in operation S140. Fig.12 of each operation, and Fig.12 The search point SP described in is a valid search point (SP_V).

[0125] The overlapping point center pixel removal unit 140 may obtain the position (X) of the point center pixel corresponding to the search point SP corresponding to the current search point index. i ,Y i )(S152).

[0126] The overlapping point center pixel removal unit 140 may obtain the positions (X) of the center pixels of each point corresponding to the adjacent search points adjacent to the search point SP corresponding to the current search point index. j ,Y j )(S153). Here, the adjacent search point may be a search point that is adjacently arranged in the upper, lower, left and right directions or in the diagonal direction relative to the search point SP corresponding to the current search point index, and may refer to a search point at which the search point SP corresponding to the current search point index and the kernel (KN) overlap with each other.

[0127] The overlapping point center pixel removal unit 140 may compare the position (X) of the point center pixel corresponding to the search point SP corresponding to the current search point index. i ,Y i ) and the position of the center pixel corresponding to the adjacent search point (X j ,Y j ), and it can be determined whether the two positions are identical to each other based on the comparison result (S154).

[0128] If the point center pixel position (X i ,Y i ) and the position of the center pixel of the point (X j ,Y j) are the same as each other (Yes in S154), the dot center pixels may overlap with each other. When the dot center pixels overlap with each other, the overlapping point center pixel removal unit 140 may remove one of the overlapping point center pixels (S155). In some implementations, the overlapping point center pixel removal unit 140 may select only the dot center pixel (DCP) corresponding to the search point with a relatively smaller or relatively larger index.

[0129] If the point center pixel position (X i ,Y i ) and the position of the center pixel of the point (X j ,Y j ) are different from each other (S154 is No), the point center pixels may not overlap with each other, and therefore, operation S155 may not be performed.

[0130] The overlapping point center pixel removing unit 140 may determine whether operation S153 has been performed on all search points adjacent to the search point SP ( S156 ).

[0131] If operation S153 is not performed on all search points adjacent to the search point SP (operation S156 is No), operation S153 may be performed on other search points adjacent to the search point SP. Thereafter, operations S153 to S156 may be repeatedly performed on other search points adjacent to the search point SP. When operations S153 to S156 are performed on all search points adjacent to the search point SP, the overlapping point center pixel removal unit 140 may remove overlapping point center pixels one by one with respect to the search point SP.

[0132] When operation S153 has been performed on all search points adjacent to the search point SP (Yes at S156) and operation S152 has not been performed on all search points (SP) (No at S157), the overlapping point center pixel removal unit 140 may increase the current search point index (i) by 1 (S158) and may return to operation S152. These operations may be repeated until operation S152 is performed on all search points. Then, if operation S152 is performed on all search points (Yes at S157), the point center pixel information of all point center pixels may be stored in the frame memory 400 to prevent the position (X) of the point center pixel from being overlapped. i ,Y i ) and amplitude value (amp i ) overlap each other.

[0133] As mentioned above Figures 9 to 12 As described, when the dot center pixel is determined by using the amplitude value as the intensity of the depth data, the dot center pixel can be determined more accurately using only the active component (actual signal component) other than the ambient light component.

[0134] Fig.13Some implementations based on the disclosed technology include Figure 4 A block diagram of another example of an imaging device 1300 with a dot-center pixel detector is shown.

[0135] Reference Fig.13 , the imaging device 1300 may include a depth image sensing device 500 and an image signal processor 600 .

[0136] The imaging device 1300 may not include a frame memory for the operation of the dot center pixel detector 100. The dot center pixel detector 100 may be included in the depth image sensing device 500 instead of the image signal processor 600. In addition, the line memory 560 and the dot data memory 570 for the operation of the dot center pixel detector 100 may be included in the depth image sensing device 500.

[0137] The depth image sensing device 500 may include a light source module 510 , a pixel array 520 , a pixel driver 530 , a timing controller 540 , a pixel readout circuit 550 , a line memory 560 , a dot data memory 570 , and a dot center pixel detector 100 . Fig.13 The functions and operations of the light source module 510, pixel array 520, pixel driver 530, timing controller 540 and pixel readout circuit 550 shown may be similar to those of FIG. Fig. 9 The illustrated light source module 510, pixel array 520, pixel driver 530, timing controller 540, and pixel readout circuit 550 are identical. Therefore, for the sake of brevity, redundant descriptions thereof will be omitted herein.

[0138] The line memory 560 may store the depth data output from the pixel readout circuit 250. The line memory 560 may have a storage capacity capable of storing the depth data corresponding to a portion of one frame. Here, a portion of one frame may correspond to pixels included in a plurality of rows of the pixel array 520, and the number of rows may be determined to be the same as the number of intervals between vertically adjacent search points (SP) to each other, or may be determined to be a larger number than the number of intervals between vertically adjacent search points (SP), but the scope or spirit of the disclosed technology is not limited thereto.

[0139] The point data memory 570 may store not only data required for the operation of the point center pixel detector 100 (eg, search point information, point center pixel information, etc.) but also instructions.

[0140] Since the dot center pixel detector 100 is included in the depth image sensing device 500 , the line memory 560 and the dot data memory 570 installed in the depth image sensing device 500 may be implemented with minimum capacity, thereby minimizing the required memory capacity.

[0141] The dot center pixel detector 100 may generate dot center pixel information using only internal resources of the depth image sensing device 500. The depth image generator 610 of the image signal processor 600 may calculate a phase difference between a modulated light signal (MLS) and a reflected modulated light signal (MLS_R) of each of a plurality of dot center pixels by referring to the dot center pixel information received from the depth image sensing device 500. Then, the depth image generator 610 may calculate a distance between the target object (OBJ) and the depth image sensing device 200 based on the calculated phase difference, and thus may generate a depth image.

[0142] Fig.14 The following are some examples of the operation of some implementations based on the disclosed technology. Fig.13 A flow chart of an example of a method of an imaging apparatus is shown. Fig.15 is a conceptual diagram illustrating an example of operations performed on a first row group based on some implementations of the disclosed technology. Fig.16 is a conceptual diagram illustrating an example of operations performed on a second row group based on some implementations of the disclosed technology. Fig.17 and Fig.18 Some implementations based on the disclosed technology are shown in Fig.14 A conceptual diagram of an example of operation S250 is shown.

[0143] Reference Fig.14 , except for some operations, Fig.14 The method of operating the imaging device is shown in Figure 5 The method of operating the point center pixel detector described in is basically the same as that described in FIG. 1 , so its redundant description will be omitted in this paper. The following description will be based on the same Figure 5 Those different characteristic operations are given as the center.

[0144] First, the depth image sensing device 500 may generate first and third depth data C0 and C2 in a first sensing period SP1, and may store the first and third depth data C0 and C2 of a current row of the pixel array 520 in the row memory 560 (S210).

[0145] When the capacity of the line memory 560 is not full (No in S212 ), the depth image sensing device 500 may repeatedly perform operation S210 .

[0146] When the capacity of the line memory 560 is full (S212 is yes), the search point management unit 110 may determine or manage search points arranged according to a predetermined rule with respect to the first depth data and the third depth data (C0, C2) corresponding to the line group consisting of the lines stored in the line memory 560 (S220).

[0147] The nearby pixel comparison unit 120 may determine a point center pixel (DCP) of each search point (SP) based on depth data of pixels located within a predetermined range of the corresponding search point SP (S230).

[0148] The valid pixel determination unit 130 may compare the intensity of the depth data of each dot center pixel determined by the nearby pixel comparison unit 120 with a valid threshold value to determine whether each dot center pixel (DCP) is a valid pixel (S240).

[0149] exist Fig.15 , the nearby pixel comparison unit 120 may determine a point center pixel (DCP1) of a point based on depth data of pixels located in a kernel (KN1) corresponding to each search point included in a first line group (L1) stored in the line memory 560.

[0150] In addition, the effective pixel determination unit 130 may compare the intensity of the depth data of each dot center pixel (DCP1) determined by the nearby pixel comparison unit 120 with the effective threshold value, and may determine whether each dot center pixel (DCP1) is a valid pixel based on the comparison result. When the dot center pixel DCP1 is a valid pixel, the search point corresponding to the corresponding dot center pixel DCP1 may be determined as a valid search point (SP_V). In addition, when the dot center pixel DCP1 is not a valid pixel, the search point corresponding to the corresponding dot center pixel DCP1 may be determined as an invalid search point (SP_I).

[0151] When operation S240 is completed, the dot center pixel detector 100 may determine whether the detection operation of the dot center pixel has been completed for all rows of the pixel array 520 ( S214 ).

[0152] When the dot center pixel detection operation for all rows of the pixel array 520 is not completed (S214 is No), the dot center pixel detector 100 may request the timing controller to read out information about the next row group, and may store the read information. The timing controller 540 may control the pixel driver 530, the pixel readout circuit 550, and the row memory 560 to perform operation S210 for the next row group. Therefore, operations S210 to S240 may be performed on the second row group L2 serving as the next row group.

[0153] exist Fig.16 In the embodiment, the nearby pixel comparison unit 120 may determine a point center pixel (DCP2) of a point based on depth data of pixels located in a kernel (KN2) corresponding to each search point included in a second line group (L2) stored in the line memory 560.

[0154] In addition, the effective pixel determination unit 130 may compare the intensity of the depth data of each dot center pixel (DCP2) determined by the nearby pixel comparison unit 120 with the effective threshold value, and may determine whether each dot center pixel (DCP2) is a valid pixel based on the comparison result. When the dot center pixel DCP2 is a valid pixel, the search point corresponding to the corresponding dot center pixel (DCP2) may be determined as a valid search point (SP_V). In addition, when the dot center pixel DCP2 is not a valid pixel, the search point corresponding to the dot center pixel (DCP2) may be determined as an invalid search point (SP_I).

[0155] When the detection operation of the dot center pixels is completed for all rows of the pixel array 520 (Yes in S214 ), the overlapping dot center pixel removal unit 140 may remove the overlapping dot center pixels from among the dot center pixels used as valid pixels ( S250 ).

[0156] exist Fig.17 In the embodiment, for the convenience of description, it is assumed that the number of row groups (L1 to L8) corresponding to the pixel array 520 is set to 8, but the scope of the disclosed technology is not limited thereto. Each dot center pixel (DCP1, DCP2) may correspond to each dot (DOT), and the positions of the dot center pixels (DCP1, DCP2) may be different from each other. This is because the kernel size of each search point may not be sufficiently ensured due to the limited capacity of the row memory 560.

[0157] exist Fig.18 In the embodiment, the overlapping point center pixel removal unit 140 may remove the overlapping point center pixel DCP2 from the point center pixels (DCP1, DCP2) used as valid pixels. Here, the overlapping point center pixel removal unit 140 may remove the point center pixel DCP2 having relatively low intensity of depth data from the point center pixels (DCP1, DCP2).

[0158] Fig.19 Some implementations based on the disclosed technology are shown in Fig.14 A flowchart of an example of operation S230 is shown.

[0159] Reference Fig.19 , the nearby pixel comparing unit 120 may initialize a search point index (i) serving as identification (ID) information of the search point SP (ie, i=1) (S231).

[0160] The nearby pixel comparison unit 120 can obtain the position (X) of the search point SP corresponding to the current search point index. i ,Y i), and the charge value (Bi) of the search point SP may be calculated (S232). The charge value (Bi) may be calculated as the sum (C0+C2) of the first depth data C0 and the third depth data C2 of the search point SP. In addition, the nearby pixel comparison unit 120 may calculate the position (X i ,Y i ) is updated to the position of the point center pixel, and the charge value (B i ) is updated to the charge value of the pixel at the center of the point.

[0161] The nearby pixel comparison unit 120 can obtain any pixel (X) in the kernel having the search point SP. i+Δx ,Y i+Δy )'s first depth data C0 and third depth data C2 (S233).

[0162] The neighboring pixel comparison unit 120 may use the pixel (X i+Δx ,Y i+Δy ) to calculate the pixel (X i+Δx ,Y i+Δy )'s charge value (B_cal) (S234).

[0163] If the average charge value (B_cal) of the search point SP is subtracted from the calculated charge value (B_cal), ave ) and the value obtained is greater than the threshold difference (D th )(S235-1 is yes), the nearby pixel comparison unit 120 may regard the calculated charge value (B_cal) as a valid charge value, and may compare the calculated charge value (B_cal) with the charge value (B_cal) of the dot center pixel (S235-2). i ).

[0164] If the average charge value (B_cal) of the search point SP is subtracted from the calculated charge value (B_cal), ave ) and the value obtained is less than or equal to the threshold difference (D th )(S235-1 is No), the nearby pixel comparison unit 120 may regard the calculated charge value (B_cal) as an invalid charge value and may return to operation S233.

[0165] Here, the average charge value (B ave ) may refer to the charge value (B) of a pixel (ie, a neighboring pixel) located within a predetermined range from the search point SP. i ) and the threshold difference (D th ) can be a value predetermined by experiment. i ) includes the component of the reflected modulated light signal (MLS_R) and the component of the ambient light, so the charge value (Bi ) may not be used as a reference value, and is only used when the difference between the calculated charge value (B_cal) and the average charge value (Bave) is greater than a predetermined threshold difference (D th ), the calculated charge value (B_cal) can be considered as a valid value so that only the component caused by the point light but not the component caused by the ambient light can be updated to the point center pixel.

[0166] If the calculated charge value (B_cal) is greater than the charge value of the pixel at the center of the point (B i )(S235-2 is yes), then the nearby pixel comparison unit 120 can compare the pixel (X i+Δx ,Y i+Δy ) and the calculated charge value (B_cal) are updated to the position of the center pixel (X i ,Y i ) and the charge value of the pixel at the center of the point (B i )(S236).

[0167] If the calculated charge value (B_cal) is less than or equal to the charge value of the center pixel (B i )(S235-2 is No), the position of the center pixel of the point can be maintained (X i ,Y i ) and charge value (B i ) without performing operation S236.

[0168] The nearby pixel comparison unit 120 may determine whether operation S233 has been performed on all pixels in the kernel including the search point SP ( S237 ).

[0169] If operation S233 is not performed on all pixels in the kernel (S237 is No), any pixel (X) in the kernel on which operation S233 has been performed may be i+Δx ,Y i+Δy ) other pixels except the kernel perform operation S233. Thereafter, operations S233 to S237 may be repeatedly performed on all pixels in the kernel. When operations S233 to S237 are performed on all pixels in the kernel, the kernel having the highest charge value (B i ) can be determined as the point center pixel, and the position of the corresponding point center pixel (X i ,Y i ) and charge value (B i ) can be stored in the point data storage 570.

[0170] If operation S233 has been performed on all pixels in the kernel (Yes at S237) and operation S232 has not been performed on all search points (No at S238), the nearby pixel comparison unit 120 may increase the current search point index (i) by 1 (S239) and may return to operation S232. These operations may be repeated until operation S232 is performed on all search points. If operation S232 is performed on all search points (Yes at S238), the position (X) of the point center pixel of each of all search points is calculated. i ,Y i ) and charge value (B i ) can be stored in the point data storage 570.

[0171] although Fig.19 An embodiment is shown in which necessary information is stored in the dot data memory 570 using the charge value (Bi) in the case of using the line memory 560 instead of the frame memory 400. However, other implementations are possible, and it should be noted that necessary information may also be stored in the dot data memory 570 by using the amplitude value (amp) in the case of using the line memory 560. That is, for example, instead of controlling all pixels of the pixel array 520 using only the first modulation control signal MCS1 and the second modulation control signal MCS2, each odd pixel among the pixels belonging to a row may receive the first modulation control signal MCS1 having a 0 degree phase difference (i.e., in phase) with respect to the modulated light signal (MLS) and the second modulation control signal MCS2 having a 180 degree phase difference with respect to the modulated light signal (MLS), and each even pixel among the pixels belonging to a row may receive the first modulation control signal MCS1 having a 90 degree phase difference with respect to the modulated light signal (MLS) and the second modulation control signal MCS2 having a 270 degree phase difference with respect to the modulated light signal (MLS). According to this control method (pseudo 4TAP), the first to fourth depth data (C0-C4) may be stored in the line memory 560, and thus the position of the center pixel (X i ,Y i ) and amplitude value (amp i ) may be stored in the point data memory 570. In this case, the first to fourth depth data (C0-C4) may be configured to correspond to two pixels instead of one pixel. This configuration may be disadvantageous in terms of resolution, but may be advantageous in that the line memory 560 may be used instead of the frame memory 400. In addition, the position of the center pixel of the generated point (X i ,Y i ) and the amplitude value of the pixel at the center of the point (amp i ) can be processed with Fig.11 The method described in is essentially the same.

[0172] Fig. 20Some implementations based on the disclosed technology are shown in Fig.14 A flow chart of an example of operation S250 is shown.

[0173] Reference Fig. 20 , the overlapping point center pixel removal unit 140 may initialize the search point index (i) used as identification (ID) information of the search point SP (ie, i=1) (S251). Here, it should be noted that the operation may be performed only for the valid search point (SP_V) determined in operation S240. Fig. 20 of each operation, and Fig. 20 The search point SP mentioned in is a valid search point (SP_V).

[0174] The overlapping point center pixel removal unit 140 may obtain the position (X) of the point center pixel corresponding to the search point SP corresponding to the current search point index. i ,Y i )(S252).

[0175] The overlapping point center pixel removal unit 140 may obtain the positions (X) of the center pixels of each point corresponding to the adjacent search points adjacent to the search point SP corresponding to the current search point index. j ,Y j )(S253). Here, the adjacent search point may be a search point adjacently disposed in the upper, lower, left and right directions or in the diagonal direction with respect to the search point SP corresponding to the current search point index.

[0176] The overlapping point center pixel removal unit 140 may calculate a pixel-to-pixel distance (l) indicating a position (x) of a point center pixel corresponding to a search point SP corresponding to a current search point index. i ,Y i ) and the position of the center pixel corresponding to the adjacent search point (X j ,Y j ) between (S253-1).

[0177] The overlapping point center pixel removal unit 140 may compare the pixel-to-pixel distance (l) with the diameter of each point (DOT) to determine whether two pixels are close to each other (S254). Here, the diameter of each point (DOT) may be a value predetermined by experiment indicating the maximum diameter of each point (DOT).

[0178] If the distance (l) between pixels (i.e., the pixel-to-pixel distance "l") is less than the diameter of each dot (DOT) (Yes at S254), the dot center pixels may correspond to one dot and may overlap each other, so that the overlapping dot center pixel removal unit 140 may remove one of the overlapping dot center pixels (S255). In some implementations, the overlapping dot center pixel removal unit 140 may select only the dot center pixel having a higher charge value from among the dot center pixels.

[0179] If the distance (1) between pixels is greater than or equal to the diameter of the dot (DOT) (No in S254), the dot center pixels may not overlap with each other, and thus, operation S255 may not be performed.

[0180] The overlapping point center pixel removing unit 140 may determine whether operation S253 has been performed on all search points adjacent to the search point SP ( S256 ).

[0181] If operation S253 is not performed on all search points adjacent to the search point SP (S256 is No), operation S253 may be performed on other search points adjacent to the search point SP. Thereafter, operations S253 to S256 may be repeatedly performed on other search points adjacent to the search point SP, and operations S253 to S256 may be performed on all search points adjacent to the search point SP. As a result, the overlapping point center pixel removal unit 140 may remove overlapping point center pixels one by one with respect to the search point SP.

[0182] If operation S253 is performed on all neighboring search points of the search point SP (Yes at S256) and operation S252 is not performed on all search points (SP) (No at S257), the overlapping point center pixel removal unit 140 may increase the current search point index (i) by 1 (S258) and may return to operation S252. These operations may be repeated until operation S252 is performed on all search points. Then, if operation S252 is performed on all search points (Yes at S257), the point center pixel information of all point center pixels may be stored in the point data storage 570 to prevent the position (X) of the point center pixel from being overlapped. i ,Y i ) and amplitude value (amp i ) overlap each other.

[0183] As mentioned above Figures 13 to 20 As described, when the charge value is used as the intensity of the depth data to determine the dot center pixel, only one frame may be needed instead of two frames, and therefore, only a line memory may be used instead of a frame memory to detect the dot center pixel.

[0184] As apparent from the above description, the depth image sensing device can efficiently and accurately detect the distance regardless of the scene, the image signal processor, and the image signal processing method.

[0185] Fig.21 It is shown that Figure 4 A block diagram of an example of a computational device for a point center pixel detector.

[0186] Reference Fig.21 , the computing device 1000 may represent a computer program product for executing Figure 4 An example of a hardware configuration for the operation of the dot-center pixel detector 100 is shown in FIG.

[0187] The computing device 1000 may be mounted on a chip included in an imaging device. According to one embodiment, the computing device 1000 may be mounted on a chip equipped with a depth image sensing device or a chip equipped with an image signal processor, but the scope of the present invention is not limited thereto.

[0188] in addition, Fig. 9 or Fig.13 The internal configuration or arrangement of the depth image sensing device and the image signal processor described in the embodiment may vary according to the implementation. For example, at least a portion of the image sensing device may be included in the image signal processor. Alternatively, at least a portion of the image signal processor may be included in the image sensing device. In this case, at least a portion of the image signal processor may be mounted together on a chip on which the image sensing device is mounted.

[0189] The computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0190] Processor 1010 can process execution Figure 4 The data and / or instructions required for the operation of the components 110 to 140 of the dot center pixel detector 100 described in FIG.

[0191] The memory 1020 may store data and / or instructions required to perform operations of the components 110 to 140 of the dot center pixel detector 100, and may be accessed by the processor 1010. For example, the memory 1020 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0192] That is, a computer program for executing the operation of the dot center pixel detector 100 disclosed in this document is recorded in the memory 1020 , executed and processed by the processor 1010 , thereby realizing the operation of the dot center pixel detector 100 .

[0193] The input / output interface 1030 is an interface that connects an external input device (eg, keyboard, mouse, touch panel, etc.) and / or an external output device (eg, display) to the processor 1010 to allow transmission and reception of data.

[0194] The communication interface 1040 is a component that can transmit and receive various data with an external device (eg, an application processor, an external memory, etc.), and may be a device that supports wired or wireless communication.

[0195] The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized through the above-mentioned patent documents.

[0196] Although a number of exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments are contemplated based on what is described and / or illustrated in this patent document.

[0197] CROSS-REFERENCE TO RELATED APPLICATIONS

[0198] This patent document claims priority to and the benefit of Korean Patent Application No. 10-2023-0085019, filed on Jun. 30, 2023, the disclosure of which is incorporated herein by reference in its entirety as a part of the disclosure of this patent document.

Claims

1. A depth image sensing device, the depth image sensing device comprising: a line memory storing depth data generated by detecting dot pattern light reflected from a scene in units of line groups; a dot center pixel detector that determines a dot center pixel having the highest intensity depth data in each of a plurality of kernels included in the row group; as well as A point data memory stores point center pixel information including the position of the point center pixel.

2. According to the depth image sensing device according to claim 1, the intensity of the depth data refers to the charge value of the first depth data and the second depth data generated by using a first modulation control signal having a predetermined phase difference relative to the dot pattern light emitted to the scene and a second modulation control signal having a phase opposite to the phase of the first modulation control signal.

3. The depth image sensing device according to claim 2, in, The charge value is the sum of the first depth data and the second depth data, and When a value obtained by subtracting an average value of charge values ​​of neighboring pixels of the corresponding pixel from the charge value is greater than a predetermined threshold difference, the charge value is regarded as a valid charge value.

4. The depth image sensing device according to claim 1, further comprising: a pixel array including a plurality of pixels for sensing the reflected dot pattern light, The number of rows included in the row group is smaller than the number of rows included in the pixel array.

5. The depth image sensing device according to claim 1, wherein: The point center pixel detector comprises: A search point management unit determines a search point included in each of the plurality of kernels.

6. The depth image sensing device according to claim 5, wherein: A distance between the search point and an adjacent search point adjacent to the search point is smaller than a distance between points of the dot pattern light emitted to the scene.

7. The depth image sensing device according to claim 1, wherein: The point center pixel detector comprises: A nearby pixel comparing unit determines the point center pixel by comparing intensities of the depth data of a plurality of pixels included in each of the plurality of kernels with one another.

8. The depth image sensing device according to claim 1, wherein: The point center pixel detector comprises: A valid pixel determination unit is configured to determine whether the point center pixel is a valid pixel by comparing the intensity of the depth data of the point center pixel with a valid threshold.

9. The depth image sensing device according to claim 8, wherein: The point center pixel detector comprises: An overlapping point center pixel removing unit removes at least one overlapping point center pixel from among the point center pixels that are valid pixels.

10. The depth image sensing device according to claim 9, wherein: The overlapping dot center pixel is a pixel spaced apart from another dot center pixel by a predetermined distance, the predetermined distance being smaller than a diameter of each dot of the reflected dot pattern light.

11. An image signal processor, the image signal processor comprising: a point center pixel detector that determines a point center pixel in each of the plurality of kernels having depth data of highest intensity; as well as A depth image generator is provided for calculating the distance to a scene using the depth data of the center pixel of the point.

12. The image signal processor according to claim 11, wherein: The intensity of the depth data refers to amplitude values ​​of first to fourth depth data generated by using first to fourth modulation control signals having different phase differences with respect to the dot pattern light emitted to the scene.

13. The image signal processor according to claim 11, wherein: The point center pixel detector comprises: A search point management unit determines a search point included in each of the plurality of kernels.

14. The image signal processor according to claim 13, wherein: A distance between the search point and an adjacent search point adjacent to the search point is smaller than a distance between points of the dot pattern light emitted to the scene.

15. The image signal processor according to claim 11, wherein: The point center pixel detector comprises: A nearby pixel comparing unit determines the point center pixel by comparing intensities of the depth data of a plurality of pixels included in each of the plurality of kernels with one another.

16. The image signal processor according to claim 11, wherein: The point center pixel detector comprises: A valid pixel determination unit is configured to determine whether the point center pixel is a valid pixel by comparing the intensity of the depth data of the point center pixel with a valid threshold.

17. The image signal processor according to claim 16, wherein: The point center pixel detector comprises: An overlapping point center pixel removing unit removes at least one overlapping point center pixel from among the point center pixels that are valid pixels.

18. The image signal processor according to claim 17, wherein: The overlapping point center pixel is a pixel having the same position as another point center pixel.

19. A method for processing an image signal, the method comprising the following steps: determining at least one point center pixel of depth data having a highest intensity in each of the plurality of kernels; Determine whether the point center pixel is a valid pixel by comparing the intensity of the depth data of the point center pixel with a valid threshold; removing overlapping point center pixels from the point center pixels as valid pixels; as well as The depth data of the center pixel of the point is used to calculate the distance to the scene.

20. The image signal processing method according to claim 19, wherein: The valid pixel is determined by repeatedly performing the step of determining the dot center pixel of the depth data having the highest intensity and the step of determining whether the dot center pixel is the valid pixel.

Images