Image processing apparatus and image processing method

Abstract

The invention relates to an image processing apparatus and an image processing method. The image processing device comprises a determining unit and a processing unit, the determination unit is configured to i) determine, for a pixel group including a pixel in a pixel array of an imaging device that receives reflected light from an object carrying a dot pattern, a pixel signal of the pixel based on a maximum amplitude value among amplitude values of pixel signals of the pixel, determining whether to perform an averaging process on at least some of the pixels included in the pixel group corresponding to the dot pattern, and ii) determining a target pixel from among the pixels based on a result of summing at least some of the amplitude values of the pixel signals of the pixels in the pixel group; and a calculation unit in communication with the determination unit and configured to perform an averaging process on a pixel signal of the target pixel to calculate a distance value of a distance between the imaging device and the object corresponding to the dot pattern.

Description

Technical Field

[0001] The disclosure of this patent document relates to apparatus and methods for processing images. Background Technology

[0002] Image sensing devices are used to capture optical images by utilizing the properties of photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communications industries, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robotics, security cameras, and medical miniature cameras.

[0003] Today, image sensing devices are not only actively used to acquire color images, but also to sense the distance to objects. The distance between an object and an image sensing device can be measured based on time-of-flight (ToF) measurements, which directly or indirectly measure the time it takes for an emitted light pulse to reach the object and return to the image sensing device. Summary of the Invention

[0004] Some implementations of the disclosed technology have been developed to address the problems in the prior art, while fully preserving the advantages of the prior art implementation.

[0005] One aspect of the disclosed technology provides an image processing apparatus for performing averaging processing on pixels corresponding to a dot pattern.

[0006] One aspect of the disclosed technology provides an image processing apparatus that determines whether to perform averaging processing on pixels based on whether to improve the accuracy of distance measurement.

[0007] One aspect of the disclosed technology provides an image processing apparatus that performs averaging on at least some pixels by determining the pixels to which averaging will be performed.

[0008] One aspect of the disclosed technology provides an image processing apparatus that determines whether to perform averaging processing by using the signal-to-noise ratio.

[0009] One aspect of the disclosed technology provides an image processing apparatus that determines a target pixel to which averaging processing will be performed by using a signal-to-noise ratio.

[0010] The technical problems to be solved by the disclosed technology are not limited to those mentioned above. Those skilled in the art to which this disclosure pertains will clearly understand from the following description any other technical problems not mentioned herein.

[0011] In one aspect, the image processing apparatus may include: a determining unit configured to i) determine, for a pixel group comprising pixels in a pixel array of an imaging apparatus that receives reflected light from an object carrying a dot pattern, whether to perform averaging processing on at least some of the pixels in the pixel group corresponding to the dot pattern based on the maximum amplitude value among the amplitude values ​​of the pixel signals of the pixels, and ii) determine a target pixel from the pixels based on the result of summing at least some of the amplitude values ​​of the pixel signals of the pixels in the pixel group; and a calculation unit communicating with the determining unit and configured to perform averaging processing on the pixel signals of the target pixel to calculate a distance value between the imaging apparatus and the object corresponding to the dot pattern.

[0012] According to an implementation, the determining unit is configured to determine to perform the averaging process in response to the maximum amplitude value being greater than a first threshold amplitude value and less than a second threshold amplitude value.

[0013] According to an implementation, the determining unit is configured to determine the first threshold amplitude value by using a first sum amplitude value required to perform averaging on all pixels and a second sum amplitude value obtained by normalizing and summing the amplitude values ​​of the pixel signals.

[0014] According to an implementation, the determining unit is configured to determine the amplitude value required to calculate the distance value by using one of the pixels in the pixel group as the second threshold amplitude value.

[0015] According to an implementation, the determining unit is configured to determine the first threshold amplitude value and the second threshold amplitude value based on the use of dark noise and signal-to-noise ratio.

[0016] According to an implementation, the determining unit is configured to determine the target pixel by performing a plurality of summation stages that sum the amplitude values ​​one by one in descending order from the maximum amplitude value to the minimum amplitude value, and comparing the sum calculated in each summation stage with a threshold sum value.

[0017] According to an implementation, the determining unit is configured to determine the pixel corresponding to the target amplitude value that is summed until the calculated sum is greater than or equal to the threshold sum value as the target pixel.

[0018] According to an embodiment, the determining unit is configured to determine all pixels included in the pixel group as the target pixel in response to a value obtained by adding all amplitude values ​​of the pixel signals included in the pixel group being less than the threshold sum value.

[0019] According to an implementation, the determining unit is configured to determine the threshold summation value as having different values ​​depending on the number of target amplitude values ​​to be summed in each summation stage.

[0020] According to an implementation, the determining unit is configured to determine the total threshold value based on dark noise and signal-to-noise ratio.

[0021] According to an embodiment, the determining unit is configured to further determine the pixel group such that the amplitude value of the center pixel of the pixel group is the maximum amplitude value.

[0022] According to an embodiment, the computing unit is configured to calculate the distance value by performing the averaging process based on the amount of charge generated by the target pixel.

[0023] According to one aspect of this disclosure, the image processing apparatus may include: a determining unit configured to sum the amplitude values ​​of pixel signals of pixels included in a pixel group of an imaging apparatus, the imaging apparatus guiding light carrying a dot pattern to an object and receiving reflected light from the object, the summing of the amplitude values ​​including multiple summing stages to sum the pixel signals one by one in descending order and determine a target pixel from the pixels to be averaging processed based on the sum value; and a calculation unit configured to calculate a distance value between the imaging apparatus and the object corresponding to the dot pattern by performing the averaging process on the target pixel.

[0024] According to the implementation, the determining unit may determine whether to perform averaging processing on at least some of the pixels based on the maximum amplitude value among the amplitude values; and determine the target pixel based on the determination to perform the averaging processing.

[0025] According to an implementation, the determining unit may be configured to determine to perform the averaging process in response to the maximum amplitude value being greater than a first threshold amplitude value and less than a second threshold amplitude value.

[0026] According to an implementation, the determining unit is configured to determine the first threshold amplitude value by using a first sum amplitude value obtained by averaging all pixels and a second sum amplitude value obtained by normalizing and summing the amplitude values ​​of the signals of the pixels.

[0027] According to an implementation, the determining unit is configured to determine the amplitude value required to calculate the distance value by using one of the pixels in the pixel group as the second threshold amplitude value.

[0028] According to an implementation, the determining unit is configured to determine the pixel corresponding to the target amplitude value that is summed until the sum value is greater than or equal to a threshold sum value as the target pixel.

[0029] According to an implementation, the determining unit is configured to determine all pixels included in the pixel group as the target pixel in response to a value obtained by adding all amplitude values ​​being less than the threshold sum.

[0030] According to one aspect of this disclosure, the image processing method may include the following steps: determining a target pixel from the pixels to be averaging, based on the result of summing at least some amplitude values ​​of the amplitude values ​​of the pixel signals of the pixels included in the pixel group corresponding to the dot pattern; and calculating a distance value corresponding to the dot pattern by performing the averaging process using the amount of charge generated by the target pixel.

[0031] The features briefly outlined above are merely examples of aspects of the detailed description of the disclosed technology that will be described below, and do not limit the scope of the disclosed technology. Attached Figure Description

[0032] The above and other objects, features and advantages of the disclosed technology will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0033] Figure 1 This is a block diagram of an imaging apparatus based on an example implementation of the disclosed technology.

[0034] Figure 2 This is a diagram used to describe an image processing method based on an example implementation of the disclosed technology.

[0035] Figure 3 This is a flowchart illustrating an image processing method based on an example implementation of the disclosed technology.

[0036] Figure 4 This is a diagram used to describe an image processing method based on an example implementation of the disclosed technology.

[0037] Figure 5 This is a flowchart illustrating an image processing method based on an example implementation of the disclosed technology.

[0038] Figure 6 This is a diagram used to describe an image processing method based on an example implementation of the disclosed technology.

[0039] Figure 7 This is a diagram used to describe an image processing method according to an example implementation of the disclosed technology.

[0040] Figure 8This is a flowchart illustrating an image processing method based on an example implementation of the disclosed technology.

[0041] Figure 9 It is shown that... Figure 1 A block diagram of an example computing device corresponding to an image processing device. Detailed Implementation

[0042] In the following description, embodiments of the disclosed technology will be presented with reference to the accompanying drawings to facilitate implementation by those skilled in the art. However, the disclosed technology may be implemented in several different forms and is not limited to the embodiments described herein.

[0043] In describing embodiments of the disclosed technology, detailed descriptions of well-known configurations or functions will be omitted if it is determined that such detailed descriptions may obscure the essential points of the disclosed technology. Parts unrelated to the description of the disclosed technology are omitted from the accompanying drawings, and similar parts are indicated by similar reference numerals / symbols throughout the specification.

[0044] Below, we will refer to Figures 1 to 9 A detailed description of example implementations of the disclosed technology is provided.

[0045] Figure 1 This is a block diagram of an imaging apparatus according to an example embodiment of the disclosed technology.

[0046] Figure 2 This is a diagram illustrating an image processing method according to an example implementation of the disclosed technology.

[0047] The following is for reference Figure 2 right Figure 1 Please provide an explanation.

[0048] refer to Figure 1 The imaging device ID can represent a device such as a digital still camera that captures still images, a digital video camera that captures video, etc. For example, the imaging device ID can be implemented using a digital single-lens reflex camera (DSLR), a mirrorless camera, or a smartphone, but the disclosed technology is not limited thereto. For example, the imaging device ID can include a device that contains an imaging element and is capable of capturing an object and generating an image. In some implementations, the imaging device ID can be a LiDAR sensor.

[0049] The imaging device ID may include the image sensing device 100 and the image processing device 200.

[0050] The image sensing device 100 may be or include a complementary metal-oxide-semiconductor image sensor (CIS) that converts incident light into electrical signals. The image sensing device 100 may include a light source 10, a lens module 20, a light source driver 30, a pixel array 110, a sensor driver 120, a readout circuit 130, and a timing controller 140.

[0051] Light source 10 can emit emitted light EL to target object 1 in response to a modulated light signal applied from light source driver 30. Light source 10 can be or includes at least one of a laser diode (LD), light-emitting diode (LED), near-infrared laser (NIR), point light source, or a monochromatic light source combining a white lamp and a monochromator, or other laser light sources that emits light in a specific wavelength band (e.g., near-infrared, infrared, or visible light). For example, light source 10 can emit light in the infrared band with wavelengths in the range of 800 nm to 1000 nm. In some implementations, the light emitted from light source 10 can be pulsed light having a preset frequency, preset period, preset amplitude, and preset pulse width. Figure 1 For ease of description, only a single light source 10 is shown, but multiple light sources can be arranged around the lens module 20.

[0052] According to an embodiment, the light source 10 may be or include a point light source that concentrates its light emission onto multiple points. The point light source can illuminate point light at multiple points by combining an optical system such as a lens or diffractive optical element (DOE) with a laser diode. Because the light spot formed by the point light source has a somewhat scalable profile due to optical limitations, it can emit light in a shape spanning multiple pixels (PX).

[0053] Lens module 20 can collect light RL reflected from target object 1 so as to focus it on pixel PX of pixel array 110. For example, lens module 20 may include a focusing lens or any other cylindrical optical element on a glass or plastic surface. Lens module 20 may include multiple lenses aligned about the optical axis.

[0054] The light source driver 30 can generate a modulated optical signal MLS for driving the light source 10 under the control of the timing controller 140, and specifically, can control the waveform (e.g., frequency, period, amplitude and pulse width) of the emitted light EL output from the light source 10.

[0055] Pixel array 110 may include a plurality of pixels PX arranged continuously in a two-dimensional matrix structure (e.g., continuously arranged in the column direction and / or row direction). Based on the control of sensor driver 120, each of the plurality of pixels PX can sense incident light incident through lens module 20 to generate a pixel signal. Pixel array 110 may include a color filter array (CFA), wherein the color filters are arranged based on a given pattern (e.g., a Bayer pattern, a quad-Bayer pattern, a nine-Bayer pattern, or an RGBW pattern) to enable the sensing of light in a preset waveform frequency band. The pattern of image data IDATA can be defined according to the type of pattern of the CFA.

[0056] In some implementations, each pixel PX can be an infrared pixel that generates a pixel signal by sensing incident light, including reflected light RL that is incident after emitted light EL from light source 10 is reflected from target object 1. For example, an infrared pixel can be a depth pixel used to calculate the distance to target object 1. In another example, an infrared pixel can include a pixel that generates an infrared image by sensing infrared light incident from the scene rather than reflected light. In some other implementations, pixel PX can include a pixel that generates a color image by sensing visible light incident from the scene. Hereinafter, the description will be based on the assumption that each pixel PX is a 2-TAP pixel used to detect the distance to target object 1 in an indirect ToF manner. A 2-TAP pixel can refer to a pixel structure with two charge accumulation gates (or “tap”) designed for indirect ToF (iTOF) depth sensing.

[0057] The sensor driver 120 can drive the pixels PX of the pixel array 110 in response to a timing signal output from the timing controller 140. For example, the sensor driver 120 can generate a control signal that can select and control the pixels PX included in at least one of the multiple row lines of the pixel array 110.

[0058] Under the control of the timing controller 140, the readout circuit 130 can generate and store image data IDATA for detecting the distance of the target object 1 by processing the pixel signals output from the pixel array 110. The image data IDATA can be digital data obtained by performing analog-to-digital conversion on the analog pixel signals. For this purpose, the readout circuit 130 may include a correlation dual sampler (CDS) for performing correlation dual sampling on the pixel signals output from the pixel array 110. Furthermore, the readout circuit 130 may include an analog-to-digital converter for converting the output signal from the correlation dual sampler into a digital signal. Additionally, the readout circuit 130 may include a buffer memory for temporarily storing the pixel data output from the analog-to-digital converter, and output the temporarily stored pixel data to the outside under the control of the timing controller 140. Simultaneously, two column lines can be provided for each column of the pixel array 110 to transmit pixel signals, and the components for processing the pixel signals output from each column line can be configured corresponding to each column line.

[0059] The timing controller 140 can generate timing signals for controlling the operation of the light source driver 30, the sensor driver 120, and the readout circuit 130. According to an embodiment, the timing controller 140 can generate timing signals based on a given set value and / or according to a request from the image processing device 200. According to an embodiment, the timing controller 140 may include logic control circuitry, a phase-locked loop (PLL) circuit, a timing control circuit, a communication interface circuit, etc.

[0060] The image processing apparatus 200 can communicate with the image sensing apparatus 100 to receive image data IDATA from the image sensing apparatus 100. The image processing apparatus 200 can generate processed image data by performing at least one image signal processing on the image data IDATA.

[0061] The image processing apparatus 200 can reduce noise in image data IDATA and perform image signal processing to improve image quality, such as demosaicing, defect pixel correction, gamma correction, color filter array interpolation, color matrix adjustment, color correction, color enhancement, or lens distortion correction. In some implementations, the image processing apparatus 200 can generate an image file by performing compression processing on the image data that has undergone image signal processing to improve image quality, or it can recover image data from an image file. The image compression format can be a reversible or irreversible format. As examples of compression formats, the JPEG (Joint Photo Experts Group) format or the JPEG 2000 format can be used for still images. In the case of moving images, moving image files can be generated by compressing multiple frames conforming to the MPEG (Moving Picture Experts Group) standard.

[0062] The image processing device 200 may be a computing device mounted on a chip independent of the chip on which the image sensing device 100 is mounted, but the disclosed technology is not limited thereto. The chip on which the image sensing device is mounted and the chip on which the image processing device 200 is mounted can communicate with each other through a given interface. According to embodiments, the chip on which the image sensing device is mounted and the chip on which the image processing device 200 is mounted can be implemented in a single package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0063] The image processing apparatus 200 may include a determining unit 210 and a calculating unit 220. The determining unit 210 and the calculating unit 220 are elements of the image processing apparatus 200 that perform specific functions, and are implemented as logic and electrical circuits / components to perform specific functions.

[0064] The determining unit 210 can also determine whether to perform averaging processing on the pixels included in the pixel group corresponding to the dot pattern.

[0065] When a point light source is used to concentrate light emission at multiple points, the light emitted by the point light source in a dot pattern can be reflected from an object and can be incident on pixels. The incident light of the dot pattern reflected from the object can have an extensible profile. Therefore, the light of the dot pattern can be emitted in a shape spanning multiple pixels. For example, see reference... Figure 2 Because dot-patterned light has an scalable contour, it can illuminate across multiple pixels (e.g., pixels 0 to 15) in the shape of the dot pattern. Figure 2 In the example shown, the light from the dot pattern illuminates across 16 pixels. (Compared to...) Figure 2 Unlike other light sources, light from a dot pattern can illuminate across a pixel group corresponding to a 3×3 or 5×5 dot pattern, but is not limited to this. The multiple pixels of the dot pattern illuminated by the light are called a "pixel group" corresponding to the dot pattern.

[0066] The merging method, as an averaging process, can be used to improve the accuracy of distance measurement by using pixels in a pixel group. When merging is performed, the signal components can be amplified, and the signal-to-noise ratio (S / N) can increase. This can mean improved accuracy of distance measurement. However, since the accuracy of distance measurement is not improved in all cases of performing averaging, the determining unit 210 can determine whether to perform averaging on pixels included in a pixel group corresponding to the dot pattern. Specifically, when the accuracy of distance measurement can be improved by averaging, the determining unit 210 can determine to perform averaging on the pixel group. For example, assuming averaging is performed on two pixels (e.g., a first pixel and a second pixel), the S / N may not improve even if averaging is performed on the first pixel and the second pixel when the amplitude value of the signal of the first pixel is relatively low. The S / N may also not improve even when the amplitude value of the signal of the second pixel is relatively low compared to the amplitude value of the signal of the first pixel. Therefore, since averaging is not needed when the S / N is not improved, the determining unit 210 according to the example embodiment of the disclosed technology can selectively perform averaging by determining whether to perform averaging.

[0067] Based on the maximum amplitude value among the signal amplitude values ​​of the pixels included in the pixel group corresponding to the dot pattern, the determining unit 210 can determine whether to perform averaging processing on at least some of the pixels.

[0068] Assuming the image sensing device 100 emits a first modulated light and drives two taps, the charge of the pixel generated by the reflected light of the first modulated light can be distributed synchronously to the floating diffusion nodes of the two taps. In this case, the amount of charge distributed can be referred to as "S0" and "S180", respectively. Furthermore, the image sensing device 100 can emit a second modulated light obtained by shifting the emission timing of the first modulated light by up to 1 / 4 period; the amount of charge obtained by the same method can be referred to as "S90" and "S270", respectively. The amplitude value of the pixel signal based on the amount of charge can be determined by Equation 1 below.

[0069] [Equation 1]

[0070] .

[0071] When the maximum amplitude value is greater than a first threshold amplitude value and less than a second threshold amplitude value, the determining unit 210 can determine that averaging processing should be performed on at least some of the pixels in the pixel group. The determining unit 210 can determine the first threshold amplitude value by using a first sum amplitude value required to perform averaging processing on all pixels included in the pixel group and a second sum amplitude value obtained by normalizing and summing the amplitude values ​​of the pixels in the pixel group. Furthermore, the determining unit 210 can determine the second threshold amplitude value as the amplitude value required when calculating a distance value using one pixel from the pixel group. Additionally, the determining unit 210 can determine the first and second threshold amplitude values ​​by using dark noise and the signal-to-noise ratio required for the desired accuracy of the distance measurement. How the determining unit 210 determines whether to perform averaging processing will be described in detail later.

[0072] When determining unit 210 determines to perform averaging, it can perform averaging on at least some pixels in the pixel group. Furthermore, determining unit 210 can determine the target pixel from the pixels of the pixel group to which averaging will be performed. For example, determining unit 210 can determine the target pixel from the pixels of the pixel group based on the result of summing at least some of the amplitude values ​​of the signals of the pixels in the pixel group. Specifically, determining unit 210 can determine the target pixel by summing the amplitude values ​​one by one in descending order from the maximum amplitude value to the minimum amplitude value and comparing the sum calculated in each summing stage with a threshold sum value. For example, determining unit 210 can determine the pixel corresponding to the target amplitude value from which the summing continues until the calculated sum value is greater than or equal to the threshold sum value as the target pixel. Furthermore, when the value obtained by summing all the amplitude values ​​of the signals of the pixels included in the pixel group is less than the threshold sum value, determining unit 210 can determine all pixels included in the pixel group as target pixels. The determination unit 210 can set the threshold summation value differently depending on the number of target amplitude values ​​to be summed during the summation phase. Furthermore, the determination unit 210 can determine the threshold summation value by considering the signal-to-noise ratio required for the desired accuracy of the distance measurement, using dark noise. Additionally, the determination unit 210 can determine the target pixel by determining whether the amplitude value of the signal of the pixel to be used for averaging is less than a given value. In other words, if it is necessary to sum amplitude values ​​less than a given value during the summation of amplitude values ​​arranged in descending order, the determination unit 210 can stop the summation process and determine the pixels involved in the summation process as the target pixels. How the determination unit 210 determines the target pixel will be described in detail later.

[0073] The calculation unit 220 can calculate the distance value by performing an averaging process. Specifically, the calculation unit 220 can calculate the distance value corresponding to the dot pattern by performing an averaging process on the target pixels determined by the determining unit 210. For example, the calculation unit 220 can calculate the distance value by performing an averaging process using the amount of charge generated from the target pixels. How the calculation unit 220 calculates the distance value will be described in detail later.

[0074] Figure 3 This is a flowchart illustrating an image processing method according to an example implementation of the disclosed technology.

[0075] Figure 4 This is a diagram used to describe an image processing method according to an example implementation of the disclosed technology.

[0076] The following is for reference Figure 4 right Figure 3 Please provide an explanation.

[0077] refer to Figure 3 At step S310, the image processing method according to an example embodiment of the disclosed technology can obtain the maximum amplitude value among the amplitude values ​​of the pixel signals from the pixels. The image processing method can set up pixel groups based on the light incident on the dot pattern onto the pixels. For example, the dot size of the dot pattern is known to the module; therefore, the image processing method can extract the pixels onto which the light of the dot pattern is incident sequentially, for example, starting from the upper left of the pixel. Pixel extraction can be based on the dot size (e.g., 3×3, 4×4, or 5×5). When the amplitude value of the signal of the center pixel among the extracted pixels is the maximum value, the image processing method can set the pixels as a pixel group. For example, as... Figure 4 As shown, when the amplitude value Amp4 of the center pixel placed at coordinates (0, 0) is the largest among the amplitude values ​​Amp0 to Amp8 of the pixel signals of the pixels included in the pixel group, the image processing method can set a 3×3 pixel group. In this case, the center pixel placed at (0, 0) can be the pixel that generates the signal with the maximum amplitude value. Therefore, the image processing method can obtain the maximum amplitude value from the center pixel of the pixel group.

[0078] At S320, the image processing method can determine whether the maximum amplitude value is greater than a first threshold amplitude value and less than a second threshold amplitude value. The image processing method can determine the first threshold amplitude value and the second threshold amplitude value by using dark noise and the signal-to-noise ratio required for the desired accuracy of the distance measurement. When the signal-to-noise ratio required for the desired accuracy of the distance measurement is called "S / N" and the dark noise is called "N", the sum amplitude value Str of the signals of the pixels required to perform averaging processing on "n" pixels can be determined by the following equation.

[0079] [Equation 2]

[0080] .

[0081] The first threshold amplitude value can be determined using the summed amplitude value Str described above. Since the outline of the light illuminating the dot pattern can be measured in advance, the summed amplitude value Str is obtained by normalizing the amplitude values ​​of the signals from the pixels of the pixel group illuminating the dot pattern to the maximum amplitude value and summing the normalized amplitude values. For example, when assuming as... Figure 4 When setting the pixel group as shown, the summation amplitude value Str obtained by summing the normalized amplitude values ​​can be determined by the following equation 3.

[0082] [Equation 3]

[0083] .

[0084] Since Stn is a predetermined value measured, the sum amplitude value Str required to perform averaging on all pixels included in the pixel group can be predicted by multiplying the maximum amplitude value by Stn. The sum amplitude value Stp required to perform averaging on all pixels included in the pixel group can be determined using Equation 4 below.

[0085] [Equation 4]

[0086] .

[0087] Here, Amp_dot_c is the maximum amplitude value. Therefore, Amp_dot_c represents the amplitude value of the signal of the center pixel placed at the center of the pixel group.

[0088] The desired maximum amplitude value, Amp_dot_c, can be predicted based on the sum amplitude value Str required to perform averaging on all pixels included in the pixel group corresponding to the dot pattern. When the predicted Amp_dot_c is denoted as Amp_dot_c_p, Amp_dot_c_p can be determined by the following Equation 5.

[0089] [Equation 5]

[0090] .

[0091] By using Amp_dot_c_p, determined by Equation 3 above, as the first threshold intensity value to serve as the lower limit of the maximum intensity value, the image processing method can prevent or reduce unnecessary processing, such as performing averaging even when the required accuracy of distance measurement has not yet been achieved.

[0092] In some implementations, because deviations between the actual light-emitting optical system and the light-receiving optical system may cause the light profile of the actual dot pattern to not perfectly match the pre-measured profile, the image processing method may not directly use Amp_dot_c_p as the first threshold amplitude value. In some implementations, the image processing method can modify Amp_dot_c_p and use the modified value as the first threshold amplitude value. For example, the image processing method can use a value that is a predetermined amount smaller than Amp_dot_c_p as the first threshold amplitude value.

[0093] As described above, the image processing method can determine the first threshold amplitude value by using the sum amplitude value required to perform averaging on all pixels included in the pixel group and the sum amplitude value obtained by normalizing and summing the amplitude values ​​of the signals of the pixels included in the pixel group.

[0094] Since the image processing method uses the Str value corresponding to "n=1" as the second threshold amplitude value, it can be determined that when the maximum amplitude value is greater than the second threshold amplitude value, a sufficient signal-to-noise ratio can be obtained even without averaging processing, using only the maximum amplitude value. Therefore, the image processing method can determine the second threshold amplitude value as the amplitude value required to calculate the distance value using a pixel from the pixel group.

[0095] When the maximum amplitude value is greater than the first threshold amplitude value and less than the second threshold amplitude value, at S330, the image processing method can perform averaging on the pixels.

[0096] When the maximum amplitude value is less than the first threshold amplitude value or greater than the second threshold amplitude value, at S340, the image processing method can calculate the distance value without performing averaging on the pixels.

[0097] Image processing methods can determine that if the maximum amplitude value is less than a first threshold amplitude value, then even if averaging is performed, the signal-to-noise ratio, the accuracy of distance measurement, etc., will not be improved.

[0098] When the maximum amplitude value is greater than the second threshold amplitude value, the image processing method can skip the averaging process, because sufficient distance measurement accuracy can be obtained by using only the maximum amplitude value.

[0099] Figure 5 This is a flowchart illustrating an image processing method according to an example implementation of the disclosed technology.

[0100] Figure 6 This is a diagram used to describe an image processing method according to an example implementation of the disclosed technology.

[0101] Figure 7 This is a diagram used to describe an image processing method according to an example implementation of the disclosed technology.

[0102] The following is for reference Figure 6 and Figure 7 right Figure 5 Please provide an explanation.

[0103] refer to Figure 5 At S510, the image processing method according to an example embodiment of the disclosed technology can determine to perform averaging processing. For example, based on the maximum amplitude value among the amplitude values ​​of the pixel signals of the pixels included in the pixel group corresponding to the dot pattern, the image processing method can determine to perform averaging processing on at least some of the pixels.

[0104] At S520, the image processing method can sum the amplitude values ​​of the signals of the pixels included in the pixel group one by one in descending order from the maximum amplitude value to the minimum amplitude value.

[0105] At S530, the image processing method can compare the sum calculated in each summing stage with the threshold sum.

[0106] At S540, the image processing method can determine the target pixel to which averaging processing will be performed based on the comparison result of comparing the calculated sum with the threshold sum.

[0107] In some implementations, the image processing method can determine the target pixel to be averaged by summing amplitude values ​​one by one in descending order from the maximum amplitude value to the minimum amplitude value and comparing the sum calculated in each summing stage with a threshold sum value. For example, the image processing method can determine the pixel corresponding to the target amplitude value that is summed until the calculated sum value is greater than or equal to the threshold sum value as the target pixel. Specifically, when the sum calculated in the first summing stage is less than the threshold sum value, the image processing method can add the amplitude values ​​of the next summing stage that conform to the descending order to the sum value. Furthermore, when the sum calculated in the second summing stage is less than the threshold sum value, the image processing method can add the amplitude values ​​of the next summing stage that conform to the descending order to the sum value. In addition, when the sum calculated in the third summing stage is greater than or equal to the threshold sum value, the image processing method can determine the pixel corresponding to the target amplitude value used for summing as the target pixel.

[0108] For reference Figure 6 The example shown describes summing amplitude values ​​to determine the target pixel. Figure 6In the image, the center pixel at coordinate (0, 0) has a pixel signal with a maximum amplitude of 15, the pixel at coordinate (1, 0) has a second maximum amplitude of 8, the pixel at coordinate (0, 1) has a third maximum amplitude of 7, and the pixel at coordinate (-1, 0) has a fourth maximum amplitude of 6. For example, refer to... Figure 6 The image processing method can perform a first summation stage to generate a sum value 23 by adding the amplitude value of 15 (the center pixel with the maximum amplitude value) to the amplitude value of 8 (the pixel at coordinates (1, 0) with the second maximum amplitude value). When the threshold sum value is set to 35, the image processing method can proceed to the next summation stage because the sum value 23 is less than 35. Therefore, the image processing method can perform a second summation stage to generate a sum value 30 by adding the amplitude value of 7 (the pixel at coordinates (0, 1) with the sum value 23. Because the sum value 30 is less than 35, the image processing method can proceed to the next summation stage. Therefore, the image processing method can perform a third summation stage to generate a sum value 36 by adding the amplitude value of 6 (the pixel at coordinates (-1, 0) with the sum value 30. Because the sum value 36 is greater than 35, the summation process can be terminated. The target amplitude values ​​for summing to generate a total value of 36 are 15, 8, 7, and 6, and the target pixels corresponding to these target amplitude values ​​can be the pixels at coordinates (0, 0), (1, 0), (0, 1), and (-1, 0), respectively. In another example, when the threshold sum value is set to 50, because by... Figure 6 The sum of the amplitude values ​​of the signals of all pixels in a 3×3 matrix does not reach 50, so the image processing method can determine all pixels in the 3×3 matrix as the target pixels.

[0109] Here, the total threshold value can be a predetermined single value, or it can be a variable based on the number of pixels to be processed. For example, when there is a signal-to-noise ratio required for the desired accuracy of distance measurement, the image processing method can determine the aforementioned Str value as the total threshold value. Therefore, the total threshold value can be a variable based on the number of pixels to be processed. For example, when the desired signal-to-noise ratio (S / N) is 6 and the dark noise (N) is 1, it can be like... Figure 7 The Str value is calculated as the sum of amplitudes required to perform averaging on “n” pixels, as shown in the curve.

[0110] Image processing methods can pre-store a threshold sum value based on the number of pixels to be averaged in memory, and can read the threshold sum value corresponding to the number of pixels to be processed. The image processing method can compare the sum amplitude value of the pixels to be processed with the read threshold sum value; therefore, the image processing method can perform averaging processing according to the required accuracy of the distance measurement without over- or under-averaging.

[0111] At S550, the image processing method can calculate the distance value between the image sensing device and the object by performing an averaging process on the target pixels (e.g., pixels at coordinates (0,0), (1,0), (0,1), and (-1,0) or all pixels included in a pixel group) determined by the above method.

[0112] As described above, assuming the image sensing device emits a first modulated light and drives two taps, the charge of the pixel generated by the reflected light of the first modulated light can be distributed synchronously to the floating diffusion nodes of the two taps in sync with the driving of the two taps. The amount of charge distributed can be referred to as "S0" and "S180", respectively. Furthermore, the image sensing device can emit a second modulated light obtained by shifting the emission timing of the first modulated light by up to 1 / 4 period; the amount of charge obtained by the same method can be referred to as "S90" and "S270", respectively. The distance "D" to the object can be determined by using the amount of charge, as shown in Equation 6 below.

[0113] [Equation 6]

[0114] .

[0115] A method of averaging multiple distance data points can be considered to perform averaging. As mentioned above, because the light in a dot pattern has a contour, when simply averaging distance values, the distance values ​​of pixels with high signal-to-noise ratios may be affected by the distance values ​​of pixels with low signal-to-noise ratios. In this case, the accuracy of distance measurement may not be improved. Therefore, the image processing method according to the example implementation of the disclosed technology can perform averaging by using charge. Specifically, assuming that five pixels are processed until the sum obtained by summing the amplitude values ​​in descending order reaches a threshold sum, the distance value Da after averaging can be calculated using Equation 7 below.

[0116] [Equation 7]

[0117] .

[0118] Here, c can be the speed of light, and F can be the modulation frequency of the modulated light.

[0119] As mentioned above, the image processing method can calculate the distance value by performing an averaging process using the amount of charge generated from the target pixel, thus improving the accuracy of distance measurement.

[0120] Figure 8 This is a flowchart illustrating an image processing method according to an example implementation of the disclosed technology.

[0121] refer to Figure 8 At S801, the image processing method according to an example embodiment of the disclosed technology can calculate the amplitude value of the signal of the pixel included in the pixel group corresponding to the dot pattern.

[0122] At S802, the image processing method can obtain data associated with the number of light (denoted as "D") incident on the dot pattern on the pixel array and the pixel group corresponding to the dot pattern, that is, the data of the pixel group on which the light of a dot pattern of d×d pixels is incident.

[0123] At S803, the image processing method can pick the m-th pixel group among the detected "D" pixel groups.

[0124] At S804, the image processing method can determine whether the maximum amplitude value Amp_dot_c (i.e., the amplitude value corresponding to the center pixel in the pixel group) among the amplitude values ​​of the signals of the pixels included in the pixel group corresponding to the dot pattern is greater than the first threshold amplitude value S_thl and less than the second threshold amplitude value S_thh.

[0125] When the maximum amplitude value Amp_dot_c is less than the first threshold amplitude value S_thl or greater than the second threshold amplitude value S_thh, the image processing method can calculate the distance value in S805. In this case, the distance value can be calculated using Equation 6 above.

[0126] When the maximum amplitude value Amp_dot_c is greater than the first threshold amplitude value S_thl and less than the second threshold amplitude value S_thh, at S806, the image processing method can sort the amplitude values ​​of the signals of the pixels included in the pixel group in descending order.

[0127] In S807, the image processing method can set the maximum amplitude value Amp_dot_c, which is the first amplitude value Amp_dot (1) among the amplitude values ​​sorted in descending order, as the initial sum value Amp_t.

[0128] Image processing methods can sum the amplitude values ​​sorted in descending order one by one with the initial sum value Amp_t. For example, an image processing method can add the (n+1)th amplitude value Amp_dot(n+1) of the amplitude values ​​sorted in descending order to the initial sum value Amp_t.

[0129] At S809, the image processing method can determine whether the initial sum value Amp_t exceeds the threshold sum value S_tha.

[0130] When the initial sum value Amp_t does not exceed the threshold sum value S_tha, at S810, the image processing method can determine whether the value of "n" is equal to the value of d×d. In other words, when the value obtained by summing all amplitude values ​​of the signals of the pixels included in the pixel group does not exceed the threshold sum value S_tha, the image processing method can execute step S810.

[0131] When the value of “n” is different from the value of d×d, that is, when the value of “n” is less than the value of d×d, at S811, the image processing method can add 1 to the value of “n” and can execute step S808.

[0132] When the initial sum value Amp_t exceeds the threshold sum value S_tha, at S812, the image processing method can calculate the distance value corresponding to the point pattern by performing an averaging process on the pixels. In this case, the distance value can be calculated using Equation 7 above.

[0133] At S813, the image processing method can determine whether the value of "m" is equal to the value of "D". Based on the above description, the image processing method can perform image processing on the light illuminating all the point patterns on the pixel array.

[0134] When the value of “m” is different from the value of “D”, at S814, the image processing method can add 1 to the value of “m” and can execute step S803.

[0135] When the value of “m” equals the value of “D”, in S815, the image processing method can perform image processing on the next frame.

[0136] Figure 9 It is shown that... Figure 1 A block diagram of an example computing device corresponding to an image processing device.

[0137] refer to Figure 9 The computing device 1000 can be shown as being used for execution Figure 1 An implementation of the hardware configuration for operating the image processing device 200.

[0138] The computing device 1000 can be mounted on a chip independent of the chip on which the image sensing device is mounted. According to an embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted can be implemented using a single package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0139] The computing device 1000 may include a processor 1010, a memory 1020, an input / output interface 1030, and a communication interface 1040.

[0140] Processor 1010 can process execution references Figure 1 The image processing apparatus 200 described herein contains data and / or instructions required for the operation of its components.

[0141] The memory 1020 may store data and / or instructions required to perform operations of components of the image processing apparatus 200 and may be accessed by the processor 1010. For example, the memory 1020 may be implemented using volatile memory (e.g., DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory)) or non-volatile memory (e.g., PROM (Programmable Read-Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or flash memory).

[0142] In other words, when a computer program for performing the operation of the image processing apparatus 200 disclosed in the specification is recorded in the memory 1020 and executed and processed by the processor 1010, the operation of the image processing apparatus 200 can be realized.

[0143] The input / output interface 1030 can provide an interface for connecting the processor 1010 to external input devices (e.g., keyboard, mouse, or touch panel) and / or external output devices (e.g., display) so that data can be sent / received.

[0144] The communication interface 1040, which is a component capable of exchanging various data with external devices (e.g., application processors or external memory), can be a device capable of supporting wired or wireless communication.

[0145] The image processing apparatus according to the example embodiments of the disclosed technology can perform averaging processing on pixels corresponding to a dot pattern.

[0146] The image processing apparatus according to an example embodiment of the disclosed technology can determine whether to perform averaging processing on pixels based on whether the accuracy of distance measurement is improved.

[0147] An image processing apparatus according to an example embodiment of the disclosed technology can perform averaging processing on at least some of the pixels by determining the pixels to be used for averaging processing.

[0148] An image processing apparatus according to an example embodiment of the disclosed technology can determine whether to perform averaging processing by using the signal-to-noise ratio.

[0149] An image processing apparatus according to an example embodiment of the disclosed technology can determine the target pixel to which averaging processing will be performed by using the signal-to-noise ratio.

[0150] Those skilled in the art will understand that the effects achievable using the disclosed technology are not limited to those specifically described above, and that other advantages of the disclosed technology will become clearer from the detailed description taken in conjunction with the accompanying drawings.

[0151] In the foregoing, although the disclosed technology has been described with reference to exemplary embodiments and accompanying drawings, the disclosed technology is not limited thereto, and various modifications and changes can be made by those skilled in the art without departing from the content described in this patent document.

[0152] Priority claims and cross-references of related applications

[0153] This patent document claims the benefit of priority to Korean Patent Application No. 10-2024-0102055, filed on July 31, 2024, with the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus, the image processing apparatus comprising: The determining unit i) determines, for a pixel group comprising pixels in a pixel array of an imaging device that receives reflected light from an object carrying a dot pattern, whether to perform averaging processing on at least some of the pixels in the pixel group corresponding to the dot pattern based on the maximum amplitude value among the amplitude values ​​of the pixel signals of the pixels; and ii) determines a target pixel from the pixels based on the result of summing at least some of the amplitude values ​​of the pixel signals of the pixels in the pixel group. as well as A calculation unit communicates with the determining unit and performs averaging processing on the pixel signal of the target pixel to calculate a distance value between the imaging device and the object corresponding to the dot pattern.

2. The image processing apparatus according to claim 1, wherein, The determining unit determines to perform the averaging process in response to the maximum amplitude value being greater than the first threshold amplitude value and less than the second threshold amplitude value.

3. The image processing apparatus according to claim 2, wherein, The determining unit determines the first threshold amplitude value by using a first sum amplitude value required to perform averaging on all pixels and a second sum amplitude value obtained by normalizing and summing the amplitude values ​​of the pixel signals.

4. The image processing apparatus according to claim 2, wherein, The determining unit determines the amplitude value required to calculate the distance value by using one of the pixels in the pixel group as the second threshold amplitude value.

5. The image processing apparatus according to claim 2, wherein, The determining unit determines the first threshold amplitude value and the second threshold amplitude value based on the use of dark noise and signal-to-noise ratio.

6. The image processing apparatus according to claim 1, wherein, The determining unit determines the target pixel by performing multiple summation stages that sum the amplitude values ​​one by one in descending order from the maximum amplitude value to the minimum amplitude value, and by comparing the sum calculated in each summation stage with a threshold sum value.

7. The image processing apparatus according to claim 6, wherein, The determining unit identifies the pixel corresponding to the target amplitude value that is summed until the calculated sum is greater than or equal to the threshold sum value as the target pixel.

8. The image processing apparatus according to claim 7, wherein, The determining unit determines all pixels included in the pixel group as the target pixels in response to a value obtained by adding all amplitude values ​​of the pixel signals included in the pixel group being less than the threshold sum value.

9. The image processing apparatus according to claim 6, wherein, The determining unit determines the threshold sum value to have different values ​​depending on the number of target amplitude values ​​to be summed in each summing stage.

10. The image processing apparatus according to claim 6, wherein, The determining unit determines the total threshold value based on dark noise and signal-to-noise ratio.

11. The image processing apparatus according to claim 1, wherein, The determining unit further determines the pixel group such that the amplitude value of the center pixel of the pixel group is the maximum amplitude value.

12. The image processing apparatus according to claim 1, wherein, The calculation unit calculates the distance value by performing the averaging process based on the amount of charge generated from the target pixel.

13. An image processing apparatus, the image processing apparatus comprising: The determining unit sums the amplitude values ​​of the pixel signals of the pixels included in the pixel group of the imaging device, the imaging device guides light carrying a dot pattern to an object and receives reflected light from the object, the summing of the amplitude values ​​includes multiple summing stages to sum the pixel signals one by one in descending order and determine the target pixel to be averaged based on the sum value from the pixels; as well as A calculation unit calculates a distance value between the imaging device and the object corresponding to the dot pattern by performing the averaging process on the target pixels.

14. The image processing apparatus according to claim 13, wherein, The determining unit: Based on the maximum amplitude value among the amplitude values, determine whether to perform averaging on at least some of the pixels; and The target pixel is determined based on the averaging process performed.

15. The image processing apparatus according to claim 14, wherein, The determining unit determines to perform the averaging process in response to the maximum amplitude value being greater than the first threshold amplitude value and less than the second threshold amplitude value.

16. The image processing apparatus according to claim 15, wherein, The determining unit determines the first threshold amplitude value by using a first sum amplitude value obtained by averaging all pixels and a second sum amplitude value obtained by normalizing and summing the amplitude values ​​of the signals of the pixels.

17. The image processing apparatus according to claim 15, wherein, The determining unit determines the amplitude value required to calculate the distance value by using one of the pixels in the pixel group as the second threshold amplitude value.

18. The image processing apparatus according to claim 13, wherein, The determining unit identifies the pixel corresponding to the target amplitude value that is summed until the sum is greater than or equal to the threshold sum value as the target pixel.

19. The image processing apparatus according to claim 18, wherein, The determining unit determines all pixels included in the pixel group as the target pixel in response to the value obtained by adding all amplitude values ​​being less than the threshold sum value.

20. An image processing method, the image processing method comprising the following steps: Based on the summation of at least some amplitude values ​​of the pixel signals of the pixels included in the pixel group corresponding to the dot pattern, the target pixel to be subjected to averaging processing is determined from the pixels. as well as The averaging process is performed using the amount of charge generated from the target pixel to calculate the distance value corresponding to the dot pattern.

21. The image processing method according to claim 20, wherein, The summation is obtained by summing the amplitude values ​​of the pixel signals one by one in descending order.

Images