Image processing apparatus and method

Abstract

To provide an image processor for performing averaging processing to pixels corresponding to a dot pattern.SOLUTION: In another general aspect, there is provided an image processing device including a determiner configured to determine whether to perform an averaging process on at least a portion of pixels included in a pixel group corresponding to a dot pattern based on a maximum amplitude value among amplitudevalues of signals of the pixels, and determine a target pixel on which the averaging process is to be performed among the pixels based on a sum of the at least a portion of the amplitudevalues when the averaging process is determined to be performed, and a calculator configured to calculate a distance value corresponding to the dot pattern by performing the averaging process on the target pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical concepts of the present disclosure relate to an apparatus and method for processing images. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields, such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical microcameras.

[0003] In recent years, image sensing devices have been actively used not only to acquire color images but also to sense the distance to an object. In particular, the ToF (time of flight) method, which directly or indirectly measures the time it takes for irradiated light to reflect off the object and reach it, is widely used. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an image processing device that performs averaging processing on pixels corresponding to a dot pattern. An object of the present disclosure is to provide an image processing device that determines whether or not to perform averaging processing on pixels depending on whether or not there is an improvement in distance measurement accuracy.

[0005] An object of the present disclosure is to provide an image processing device that performs averaging processing on at least some of the pixels by determining pixels to be subjected to averaging processing.

[0006] An object of the present disclosure is to provide an image processing device that determines whether or not to perform averaging processing using a signal-to-noise ratio. An object of the present disclosure is to provide an image processing device that determines target pixels for averaging processing using a signal-to-noise ratio.

[0007] The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the following description. [Means for solving the problem]

[0008] An image processing device according to an exemplary embodiment of the present disclosure may include a decision unit that decides whether to perform averaging processing on at least some of the pixels based on the maximum intensity value among the signal amplitude values ​​of pixels included in a pixel group corresponding to a dot pattern, and if it is decided to perform averaging processing, determines target pixels among the pixels to be subjected to averaging processing based on the result of adding up at least some of the intensity values, and a calculation unit that calculates a distance value corresponding to the dot pattern by performing averaging processing on the target pixels.

[0009] According to an embodiment, the decision unit may decide to perform the averaging process if the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value.

[0010] According to one embodiment, the determination unit can determine the first threshold intensity value using a first total intensity value required to average all of the pixels and a second total intensity value obtained by normalizing and adding up the intensity values ​​of the pixel signals.

[0011] According to an embodiment, the determiner may determine an intensity value obtained when calculating a distance value using one pixel of the pixel group as the second threshold intensity value.

[0012] According to one embodiment, the determining unit can determine the first and second threshold intensity values ​​using dark noise and a signal-to-noise ratio required for the required ranging accuracy.

[0013] According to one embodiment, the determination unit sums the intensity values ​​one by one in descending order from the maximum intensity value, and compares the sum calculated at each summation step with a threshold summation value to determine the target pixel to be subjected to the averaging process.

[0014] According to an embodiment, the determination unit may determine, as the target pixel, a pixel corresponding to the target intensity value that is summed up until the calculated sum becomes equal to or greater than a threshold sum.

[0015] According to one embodiment, the determination unit may determine all of the pixels included in the pixel group as target pixels if the sum of all the signal intensity values ​​of the pixels included in the pixel group is less than a threshold sum value.

[0016] According to an embodiment, the determiner may determine the threshold summation value to have a different value depending on the number of target intensity values ​​summed in each summation step. According to one embodiment, the determining unit can determine the threshold sum value using dark noise and a signal-to-noise ratio required for the desired ranging accuracy.

[0017] According to an embodiment, the determiner may determine the pixel group such that the intensity value of the central pixel of the pixel group is the maximum intensity value. According to one embodiment, the calculation unit may calculate the distance value by performing an averaging process using the amount of charge generated by the target pixel.

[0018] An image processing device according to an exemplary embodiment of the present disclosure may include a determination unit that generates a summation value for each summation step by summing the signal intensity values ​​of pixels included in a pixel group corresponding to a dot pattern one by one in descending order, and determines a target pixel among the pixels to be subjected to averaging processing based on the summation value, and a calculation unit that calculates a distance value corresponding to the dot pattern by performing averaging processing on the target pixel.

[0019] According to one embodiment, the decision unit determines whether to perform averaging processing on at least some of the pixels based on the maximum intensity value among the intensity values, and if it is determined whether to perform averaging processing, it can determine a target pixel.

[0020] According to an embodiment, the decision unit may decide to perform the averaging process if the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value.

[0021] In one embodiment, the first threshold intensity value can be determined using a first total intensity value required to average all of the pixels and a second total intensity value obtained by normalizing and adding together the intensity values ​​of the pixel signals.

[0022] According to one embodiment, the second threshold intensity value may be determined as an intensity value that is required when calculating a distance value using one pixel of the pixel group. According to an embodiment, the determination unit may determine, as the target pixel, a pixel corresponding to the target intensity value that is summed up until the sum becomes equal to or greater than a threshold sum value.

[0023] According to an embodiment, the determination unit may determine all of the pixels included in the pixel group as the target pixel if the sum of all the intensity values ​​is less than a threshold sum.

[0024] An image processing method according to an exemplary embodiment of the present disclosure may include a step of determining a target pixel among the pixels to be subjected to averaging processing based on the result of summing at least a portion of the intensity values ​​of the signals of pixels included in a pixel group corresponding to a dot pattern, and a step of calculating a distance value corresponding to the dot pattern by performing averaging processing using the amount of charge generated by the determined target pixel. The above briefly summarized features of the present disclosure are illustrative aspects of the detailed description of the present disclosure that follows and are not intended to limit the scope of the present disclosure. [Effects of the Invention]

[0025] The image processing device according to the exemplary embodiment of the present disclosure can perform averaging processing on pixels corresponding to the dot pattern. The image processing device according to the exemplary embodiment of the present disclosure can determine whether to perform averaging processing on pixels depending on whether or not there is an improvement in distance measurement accuracy.

[0026] The image processing device according to the exemplary embodiment of the present disclosure can perform averaging processing on at least some of the pixels by determining pixels to be subjected to averaging processing.

[0027] The image processing device according to the exemplary embodiment of the present disclosure can determine whether to perform averaging processing using the signal-to-noise ratio. The image processing device according to the exemplary embodiment of the present disclosure can determine the target pixel for averaging processing using the signal-to-noise ratio.

[0028] The effects obtained by the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the following description. [Brief explanation of the drawings]

[0029] [Figure 1]FIG. 1 is a block diagram of an imaging device according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 5] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 8] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 2 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0030] Hereinafter, with reference to the accompanying drawings, detailed descriptions of embodiments of the present disclosure will be given so that those skilled in the art can easily implement the present disclosure, however, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0031] In describing embodiments of the present disclosure, if a detailed description of known configurations or functions is deemed to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, in the drawings, parts that are not related to the description of the present disclosure will be omitted, and similar parts will be designated by similar reference numerals.

[0032] In this disclosure, when a component is "coupled," "coupled," or "connected" to another component, it means not only a direct connection, but also an indirect connection where there is another component between them. Furthermore, when a component "includes" or "has" another component, it does not mean that the other component is excluded, and that the component may further include the other component, unless otherwise specified.

[0033] In this disclosure, terms such as first and second are used only to distinguish one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0034] In this disclosure, components that are distinguished from one another are used to clearly describe the characteristics of each component and do not necessarily mean that the components are separate. That is, multiple components may be integrated and consist of a single hardware or software unit, or a single component may be distributed and consist of multiple hardware or software units. Therefore, even if not otherwise specified, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0035] In this disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of this disclosure. Furthermore, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of this disclosure.

[0036] In this disclosure, expressions of positional relationships used in this specification, such as upper, lower, left side, right side, etc., are described for the convenience of explanation, and when the drawings shown in this specification are viewed upside down, the positional relationships described in this specification may be interpreted in reverse.

[0037] In this disclosure, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" may include any one or all possible combinations of the items listed along with the corresponding phrase.

[0038] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to FIGS. FIG. 1 is a block diagram of an imaging device according to an exemplary embodiment of the present disclosure. FIG. 2 is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure.

[0039] FIG. 1 will be described below with reference to FIG. Referring to FIG. 1, an imaging device (ID) may refer to a device such as a digital still camera that captures still images or a digital video camera that captures video. For example, the imaging device ID may be implemented by, but is not limited to, a digital single lens reflex (DSLR), a mirrorless camera, or a smartphone. The imaging device ID may also be a concept that includes a device that includes an image sensor and can capture an object and generate an image. According to one embodiment, the imaging device ID may be a lidar sensor.

[0040] The imaging device ID may include an image sensing device 100 and an image processing device 200 .

[0041] The image sensing device 100 may be a complementary metal oxide semiconductor image sensor (CIS) that converts incident light into an electrical signal. The image sensing device 100 may include a light source 10, a lens module 20, a light source driver 30, a pixel array 110, a sensor driver 120, a readout circuit 130, and a timing controller 140.

[0042] The light source 10 can irradiate the target object 1 with emitted light (EL) in response to a modulation light signal (MLS) applied from the light source driver 30. The light source 10 can be a laser diode (LD), a light-emitting diode (LED), a near-infrared laser (NIR), a monochromatic illumination source combining a point light source, a white lamp, and a monochromator, or a combination of other laser light sources, all of which emit light in a specific wavelength band (e.g., near-infrared, infrared, or visible light). For example, the light source 10 can be light in the infrared wavelength band having a wavelength of 800 nm to 1000 nm. Alternatively, the light emitted from the light source 10 can be pulsed light having a predetermined frequency, period, amplitude, and pulse width. Although only one light source 10 is shown in FIG. 1 for ease of explanation, multiple light sources can be arranged around the lens module 20.

[0043] According to one embodiment, the light source 10 may be a dot light source that concentrates light emission at multiple points. The dot light source can emit spot light at multiple points by combining a laser diode with an optical system such as a lens or a DOE (Diffractive Optical Element). The spot emitted by the dot light source has a profile with a certain degree of scalability due to optical constraints, and therefore can be emitted in a form spanning multiple pixels PX.

[0044] The lens module 20 can collect reflected light (RL) from the target object 1 and focus it onto the pixels PX of the pixel array 110. For example, the lens module 20 can include a focusing lens or other cylindrical optical element with a glass or plastic surface. The lens module 20 can include multiple lenses aligned around an optical axis.

[0045] The light source driver 30 can generate an optical modulation signal MLS for driving the light source 10 in accordance with the control of the timing controller 140, and in particular can control the waveform (frequency, period, amplitude, pulse width, etc.) of the irradiation light EL output from the light source 10.

[0046] The pixel array 110 may include a plurality of pixels PX continuously arranged in a two-dimensional matrix structure (e.g., continuously arranged in a column direction and / or a row direction). Each of the plurality of pixels PX may sense incident light incident through the lens module 20 and generate a pixel signal under the control of the sensor driver 120. The pixel array 110 may include a color filter array (CFA) in which color filters are arranged according to a certain pattern (e.g., a Bayer pattern, a quad Bayer pattern, a nona Bayer pattern, an RGBW pattern, etc.) so that each pixel can sense light of a predetermined wavelength band. The pattern of the image data IDATA may be determined depending on the type of pattern the CFA has.

[0047] Each pixel PX may be an infrared pixel that generates a pixel signal by sensing incident light, including reflected light RL, which is incident upon the target object 1 after the illumination light EL emitted from the light source 10 is reflected therefrom. According to one embodiment, the infrared pixel may be a depth pixel for calculating the distance to the target object 1. According to another embodiment, the infrared pixel may include a pixel for generating an infrared image by sensing only infrared light incident from a scene, rather than reflected light. According to still another embodiment, the pixel PX may include a pixel for generating a color image by sensing visible light incident from a scene. The following description will be given on the assumption that each pixel PX is a 2-TAP pixel for detecting the distance to the target object 1 using an indirect ToF method.

[0048] The sensor driver 120 can drive the pixels PX of the pixel array 110 in response to the timing signals output from the timing controller 140. For example, the sensor driver 120 can generate control signals capable of selecting and controlling the pixels PX included in at least one row line among a plurality of row lines of the pixel array 110.

[0049] The readout circuit 130 processes pixel signals output from the pixel array 110 under the control of the timing controller 140 to generate and store image data IDATA for detecting the distance to the target object 1. The image data IDATA may be digital data obtained by analog-to-digital conversion of analog pixel signals. To this end, the readout circuit 130 may include a correlated double sampler (CDS) for performing correlated double sampling on the pixel signals output from the pixel array 110. The readout circuit 130 may also include an analog-to-digital converter (ADC) for converting the output signal from the CDS into a digital signal. The readout circuit 130 may also include a buffer circuit for temporarily storing pixel data output from the ADC and outputting it to the outside under the control of the timing controller 140. Meanwhile, two column lines for transmitting pixel signals may be provided for each column of the pixel array 110, and a configuration for processing pixel signals output from each column line may also be provided corresponding to each column line.

[0050] The timing controller 140 may generate timing signals for controlling the operations of the light source driver 30, the sensor driver 120, and the readout circuit 130. According to an embodiment, the timing controller 140 may generate timing signals according to predetermined settings and / or requirements of the image processing device 200. According to an embodiment, the timing controller 140 may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, a communication interface circuit, etc.

[0051] The image processing device 200 is capable of performing at least one image signal processing operation on the image data IDATA to generate processed image data.

[0052] The image processing device 200 may perform image signal processing on the image data IDATA to reduce noise and improve image quality, such as demosaicing, defective pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction. The image processing device 200 may also compress image data that has undergone image signal processing for improving image quality to generate an image file, or restore image data from the image file. The image compression format may be a lossless format or a lossy format. For example, the Joint Photographic Experts Group (JPEG) format or JPEG2000 format may be used for still images. For moving images, a moving image file may be generated by compressing multiple frames according to the Moving Picture Experts Group (MPEG) standard.

[0053] The image processing device 200 may be, but is not limited to, a computing device mounted on a chip separate from the chip on which the image sensing device 100 is mounted. The chip on which the image sensing device is mounted and the chip on which the image processing device 200 is mounted may communicate with each other via a predetermined interface. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the image processing device 200 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), but the scope of the present invention is not limited thereto.

[0054] The image processing device 200 may include a determining unit 210 and a calculating unit 220 . The determination unit 210 can determine whether to perform averaging processing on pixels included in a pixel group corresponding to a dot pattern.

[0055] When a dot light source that concentrates light emission in multiple spots is used as a light source, the dot pattern light emitted from the dot light source can be reflected by an object and incident on a pixel. The dot pattern light reflected by an object and incident on the pixel can have an expansive profile. Therefore, the dot pattern light can be irradiated in a form spanning multiple pixels. For example, referring to FIG. 2, the dot pattern light can have an expansive profile and can be irradiated in a dot pattern across multiple pixels (pixel 0 to pixel 15). Unlike FIG. 2, the dot pattern light can be irradiated in a dot pattern across pixel groups corresponding to dot patterns such as 3x3 and 5x5, but is not limited thereto. The multiple pixels irradiated with the dot pattern light can be referred to as a pixel group corresponding to the dot pattern.

[0056] A binning method, which is an averaging process, can be used to improve ranging accuracy using pixels of a pixel group. When binning is performed, signal components increase, increasing the signal-to-noise ratio (S / N), thereby improving ranging accuracy. However, because ranging accuracy is not improved in all cases where averaging is performed, the decision unit 210 may determine whether to perform averaging on pixels included in a pixel group corresponding to a dot pattern. Specifically, the decision unit 210 may decide to perform averaging on a pixel group if performing averaging can improve ranging accuracy. For example, assuming that averaging is performed on two pixels (a first pixel and a second pixel), if the signal intensity value of the first pixel is significantly low, averaging the first and second pixels will not improve the signal-to-noise ratio. Furthermore, if the signal intensity value of the second pixel is significantly lower than the signal intensity value of the first pixel, the signal-to-noise ratio will not be improved. Therefore, if the signal-to-noise ratio is not improved, averaging processing is not necessary, and therefore the decision unit 210 according to the exemplary embodiment of the present disclosure can selectively perform averaging processing by deciding whether or not to perform averaging processing.

[0057] The decision unit 210 may determine whether to perform averaging processing on at least some of the pixels based on the maximum intensity value among the signal amplitude values ​​of the pixels included in the pixel group corresponding to the dot pattern.

[0058] Assuming that the image sensing device 100 emits first modulated light and drives two taps, pixel charges generated by the reflected light of the first modulated light can be distributed to the respective floating diffusion nodes in accordance with the driving of the two taps. In this case, the amounts of the distributed charges can be referred to as S0 and S180, respectively. Also, when a second modulated light is emitted with the emission timing of the first modulated light shifted by 1 / 4 period, the amounts of charges acquired in the same manner can be referred to as S90 and S270, respectively. The intensity value of the pixel signal according to the amount of charge can be determined by the following Equation 1. Equation 1

[0059] TIFF2026025957000002.tif12170

[0060] The decision unit 210 may decide to perform averaging on at least some of the pixels when the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value. The decision unit 210 may determine the first threshold intensity value using a first total intensity value required to average all of the pixels included in the pixel group and a second total intensity value obtained by normalizing and adding up the intensity values ​​of the pixel signals. The decision unit 210 may also determine the second threshold intensity value as an intensity value obtained when calculating a distance value using one pixel of the pixel group. The decision unit 210 may also determine the first threshold intensity value and the second threshold intensity value using dark noise and a signal-to-noise ratio required for the desired ranging accuracy. A more detailed description of how the decision unit 210 determines whether to perform averaging will be provided below.

[0061] If the determination unit 210 determines to perform the averaging process, it may perform the averaging process on at least some of the pixels of the pixel group. Furthermore, the determination unit 210 may determine target pixels to be averaged from among the pixels of the pixel group. For example, the determination unit 210 may determine target pixels to be averaged from among the pixels of the pixel group based on a result of summing at least some of the signal intensity values ​​of the pixels of the pixel group. Specifically, the determination unit 210 may sum the intensity values ​​one by one in descending order from the maximum intensity value and compare the sum calculated in each summing step with a threshold sum to determine target pixels to be averaged. For example, the determination unit 210 may determine, as the target pixel, a pixel corresponding to a target intensity value that is summed up until the calculated sum becomes equal to or greater than the threshold sum. Furthermore, the determination unit 210 may determine, as the target pixel, all of the pixels included in the pixel group if the sum of all the signal intensity values ​​of the pixels included in the pixel group is less than the threshold sum. The determination unit 210 may determine the threshold summation value to have a different value depending on the number of target intensity values ​​summed in the summing step. The determination unit 210 may also determine the threshold summation value using dark noise and a signal-to-noise ratio required for the desired ranging accuracy. The determination unit 210 may also determine the target pixel by determining whether the signal intensity value of the pixel used in the averaging process is below a predetermined value. That is, when summing the intensity values ​​arranged in descending order, if an intensity value below a predetermined value needs to be summed, the determination unit 210 may stop the summation process and determine the pixel that participated in the summation process as the target pixel. A more detailed description of the determination unit 210's determination of the target pixel will be provided later.

[0062] The calculation unit 220 can calculate the distance value by performing an averaging process. Specifically, the calculation unit 220 can calculate the distance value corresponding to the dot pattern by performing an averaging process on the target pixel determined by the determination unit 210. For example, the calculation unit 220 can calculate the distance value by performing an averaging process using the amount of charge generated by the target pixel. More specific details of how the calculation unit 220 calculates the distance value will be described later.

[0063] FIG. 3 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. FIG. 4 is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure.

[0064] FIG. 3 will be described below with reference to FIG. Referring to FIG. 3, an image processing method according to an exemplary embodiment of the present disclosure may obtain a maximum intensity value among the pixel signal intensity values ​​in step S310. The image processing method may first set a pixel group based on the light of a dot pattern incident on the pixel. For example, the dot size of the dot pattern is known by the module. Therefore, the image processing method may extract pixels according to the dot size (e.g., 3x3, 5x5, etc.) starting from the top left of the pixel on which the light of the dot pattern is incident. The image processing method may set a pixel in the pixel group if the signal intensity value of the center pixel of the extracted pixels is the maximum. For example, as shown in FIG. 4, the image processing method may set a 3x3 pixel group. In this case, the signal intensity value (Amp4) of the center pixel located at coordinates (0,0) may be the largest among the signal intensity values ​​(Amp0 to Amp8) of the pixels included in the pixel group. In other words, the center pixel located at coordinates (0,0) may be the pixel that generates the signal with the maximum intensity value. Therefore, the image processing method may obtain the maximum intensity value from the center pixel of the pixel group.

[0065] In step S320, the image processing method can determine whether the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value. The image processing method can determine the first and second threshold intensity values ​​using dark noise and a signal-to-noise ratio required for the desired ranging accuracy. If the signal-to-noise ratio required for the desired ranging accuracy is S / N and the dark noise is N, the total intensity value Str of the pixel signals required when averaging n pixels can be determined by the following equation 2:

[0066] TIFF2026025957000003.tif11170

[0067] The first threshold intensity value can be determined using the above Str. Since the light profile of the dot pattern can be measured in advance, the intensity values ​​of the signals of the pixels in the pixel group illuminated with the light of the dot pattern can be normalized by the maximum intensity value, and the sum of the normalized intensity values ​​can be a predetermined value as the total intensity value Stn. For example, assuming that the pixel group is set as shown in FIG. 4, the total intensity value Stn obtained by summing the normalized intensity values ​​can be determined by the following Equation 3. Equation 3

[0068] TIFF2026025957000004.tif11170

[0069] Since Stn is a value determined in advance by measurement, the total intensity value Stp required when averaging all of the pixels included in the pixel group can be predicted by multiplying the maximum intensity value by Stn. The total intensity value Stp required when averaging all of the pixels included in the pixel group can be determined by the following equation 4. Equation 4

[0070] TIFF2026025957000005.tif10170Here, Amp_dot_c is the maximum intensity value, i.e., the intensity value of the signal of the central pixel located in the center of the pixel group.

[0071] Furthermore, the maximum intensity value Amp_dot_c can be predicted from the total intensity value Str required to average all pixels included in the pixel group corresponding to the dot pattern. If this predicted Amp_dot_c is represented as Amp_dot_c_p, Amp_dot_c_p can be determined by the following equation 5. Equation 5

[0072] TIFF2026025957000006.tif11170

[0073] The image processing method employs Amp_dot_c_p determined by the above-mentioned equation 3 as the first threshold intensity value, which is the lower limit value for the maximum intensity value, thereby reducing unnecessary processing such as averaging processing even when the required ranging accuracy is not achieved.

[0074] Furthermore, due to deviations in the optical systems of both the actual light-emitting and light-receiving devices, the light profile of the actual dot pattern does not necessarily match the pre-measured profile. Therefore, the image processing method does not adopt the calculated Amp_dot_c_p as the first threshold intensity value as is, but can adopt a value that is a predetermined value smaller than Amp_dot_c_p as the first threshold intensity value.

[0075] As described above, the image processing method can determine the first threshold intensity value using the total intensity value required to average all of the pixels included in the pixel group and the total intensity value obtained by normalizing and adding up the intensity values ​​of the signals of the pixels included in the pixel group.

[0076] The image processing method may use the second threshold intensity value as the Str value when n=1, and determine that a sufficient signal-to-noise ratio is obtained using only the maximum intensity value without performing averaging if the maximum intensity value is greater than the second threshold intensity value. That is, the image processing method may determine the second threshold intensity value as the intensity value obtained when calculating a distance value using one pixel of the pixel group.

[0077] The image processing method may perform an averaging process on the pixel in step S330 if the maximum intensity value is greater than the first threshold intensity value and less than the second threshold intensity value.

[0078] Alternatively, the image processing method may calculate the distance value in step S340 without performing an averaging process on the pixel if the maximum intensity value is less than the first threshold intensity value or greater than the second threshold intensity value.

[0079] The image processing method may determine that if the maximum intensity value is smaller than the first threshold intensity value, averaging will not provide any improvement in signal-to-noise ratio, distance measurement accuracy, etc.

[0080] Furthermore, in the image processing method, if the maximum intensity value is greater than the second threshold intensity value, sufficient distance measurement accuracy can already be obtained using only the maximum intensity value, so averaging processing is not necessary.

[0081] FIG. 5 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. FIG. 6 is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. FIG. 7 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure.

[0082] FIG. 5 will be described below with reference to FIGS. 5, the image processing method according to the exemplary embodiment of the present disclosure may determine to perform averaging processing in step S510. For example, the image processing method may determine to perform averaging processing on at least some of the pixels based on the maximum intensity value among the intensity values ​​of the signals of the pixels included in the pixel group corresponding to the dot pattern.

[0083] In S520, the image processing method may add up the signal intensity values ​​of the pixels included in the pixel group one by one in descending order from the maximum intensity value. The image processing method may also compare the sum calculated in each summing step with a threshold sum in S530.

[0084] In addition, the image processing method may determine, in step S540, a target pixel to be subjected to averaging processing based on a comparison result of comparing the calculated sum value with a threshold sum value.

[0085] Specifically, the image processing method may sum the intensity values ​​one by one in descending order from the maximum intensity value, compare the sum calculated in each summing step with a threshold sum, and determine a target pixel for averaging. For example, the image processing method may determine a pixel corresponding to a target intensity value that has been summed until the calculated sum is equal to or greater than the threshold sum as the target pixel. More specifically, if the sum calculated in the first summing step is less than the threshold sum, the image processing method may proceed to the next summing step and add the next intensity value in descending order to the sum. If the sum calculated in the second summing step is less than the threshold sum, the image processing method may proceed to the next summing step and add the next intensity value in descending order to the sum. If the sum calculated in the third summing step is equal to or greater than the threshold sum, the image processing method may determine a pixel corresponding to the target intensity value to be summed as the target pixel.

[0086] For example, referring to FIG. 6 , the image processing method may generate a sum of 23 by adding the intensity value of the center pixel, 15, which is the maximum intensity value, to the intensity value of 8 of the pixel at coordinates (1,0), which is the next highest intensity value after 15. If the threshold sum is set to 35, the sum of 23 is less than 35, and therefore the method may proceed to the next summation step. That is, the image processing method may generate a sum of 30 by adding the intensity value of the pixel at coordinates (0,1), which is the next highest intensity value after 8, to the sum of 23. Because the sum of 30 is less than the threshold sum of 35, the image processing method may proceed to the next summation step. That is, the image processing method may generate a sum of 36 by adding the intensity value of the pixel at coordinates (−1,0), which is the next highest intensity value after 7, to the sum of 30. Because the sum of 36 is greater than the threshold sum of 35, the summation process may end. The target intensity values ​​added together to generate the sum 36 may be 15, 8, 7, and 6, and the target pixels corresponding to the target intensity values ​​may be pixels with coordinates (0,0), (1,0), (0,1), and (-1,0). In contrast, if the threshold sum is set to 50, the sum of the signal intensity values ​​of all the 3x3 pixels in Figure 6 does not reach 50, so the image processing method can determine all the 3x3 pixels as target pixels.

[0087] Here, the threshold sum value may be a single predetermined value or may be a variable depending on the number of pixels to be processed. For example, if there is a signal-to-noise ratio required for the desired ranging accuracy, the threshold sum value can be determined as the Str value described above. Therefore, the threshold sum value may be a variable depending on the number of pixels to be processed. For example, if the required signal-to-noise ratio S / N=6 and the dark noise N=1, the total intensity value Str required when averaging n pixels can be calculated as shown in the graph in FIG. 7.

[0088] In the image processing method, a threshold sum value corresponding to the number of target pixels to be averaged is stored in advance in a memory or the like, and the threshold sum value can be read according to the number of pixels to be processed. In the image processing method, by comparing the total intensity value of the pixels to be processed with the read threshold sum value, it is possible to perform averaging just enough to achieve the required distance measurement accuracy.

[0089] In step S550, the image processing method can calculate a distance value by performing an averaging process on the target pixel (e.g., pixel with coordinates (0,0), (1,0), (0,1), (-1,0) or all pixels included in a pixel group) determined using the above-mentioned method.

[0090] As described above, assuming that the image sensing device emits the first modulated light and drives two taps, the pixel charge generated by the reflected light of the first modulated light can be distributed to each floating diffusion node in accordance with the driving of the two taps. The amounts of the distributed charge can be referred to as S0 and S180, respectively. In addition, the second modulated light, which is obtained by shifting the emission timing of the first modulated light by 1 / 4 period, can be emitted and the amounts of charge acquired in a similar manner can be referred to as S90 and S270, respectively. The distance D to the object can be determined using the amount of charge as shown in Equation 6. Equation 6

[0091] TIFF2026025957000007.tif10170

[0092] When performing the averaging process, a method of averaging distance data can be considered. However, as described above, since the light of the dot pattern has a profile, simply averaging distance values ​​will result in the distance values ​​of pixels with a high signal-to-noise ratio being affected by the distance values ​​of pixels with a low signal-to-noise ratio, and distance measurement accuracy will not improve. Therefore, the image processing method according to an exemplary embodiment of the present disclosure can perform averaging using the amount of charge. Specifically, assuming that five pixels are processed until the sum of the intensity values ​​in descending order exceeds the threshold sum, the distance value Da after averaging can be calculated using the following Equation 7. Equation 7

[0093] TIFF2026025957000008.tif11170Here, c may be the speed of light and F may be the modulation frequency of the modulated light.

[0094] As described above, the image processing method calculates the distance value by performing an averaging process using the amount of charge generated by the target pixel, thereby improving the accuracy of distance measurement.

[0095] FIG. 8 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. Referring to FIG. 8, in step S801, the image processing method according to the exemplary embodiment of the present disclosure may calculate the intensity values ​​of the signals of pixels included in a pixel group corresponding to a dot pattern.

[0096] In step S802, the image processing method can obtain data regarding the number D of dot pattern light incident on the pixel array and the pixel group corresponding to the dot pattern, i.e., the pixel group consisting of d×d pixels on which light of one dot pattern is incident. In step S803, the image processing method may select the m-th pixel group from among the D detected pixel groups.

[0097] In step S804, the image processing method can determine whether the maximum intensity value (i.e., the intensity value of the central pixel in the pixel group) Amp_dot_c among the intensity values ​​of the signals of the pixels included in the pixel group corresponding to the dot pattern is greater than the first threshold intensity value S_thl and less than the second threshold intensity value S_thh.

[0098] If the maximum intensity value Amp_dot_c is less than the first threshold intensity value S_thl or greater than the second threshold intensity value S_thh, the image processing method may calculate a distance value in step S805, which may be calculated using Equation 6 above.

[0099] In the image processing method, if the maximum intensity value Amp_dot_c is greater than the first threshold intensity value S_thl and less than the second threshold intensity value S_thh, in step S806, the intensity values ​​of the signals of the pixels included in the pixel group can be arranged in descending order.

[0100] In step S807, the image processing method may set the maximum intensity value Amp_dot_c, which is the first intensity value Amp_dot(1), among the intensity values ​​arranged in descending order as the initial sum Amp_t.

[0101] In step S808, the image processing method may sequentially add the intensity values ​​arranged in descending order to the sum Amp_t one by one. For example, the (n+1)th intensity value Amp_dot(n+1) among the intensity values ​​arranged in descending order may be added to the sum Amp_t. In step S809, the image processing method may determine whether the sum Amp_t exceeds a threshold sum S_tha.

[0102] If the sum Amp_t does not exceed the threshold sum S_tha, the image processing method may determine whether the value n is equal to the value d×d in step S810. That is, the image processing method may perform step S810 if the sum of all the signal intensity values ​​of the pixels included in the pixel group does not exceed the threshold sum S_tha.

[0103] If the value of n is not equal to the value of d×d, that is, if the value of n is smaller than d×d, the image processing method can add 1 to the value of n in step S811, and perform step S808.

[0104] If the sum Amp_t exceeds the threshold sum S_tha, the image processing method performs an averaging process on the pixels in step S812 to calculate a distance value corresponding to the dot pattern, which can be calculated using Equation 7 above.

[0105] In step S813, the image processing method can determine whether the m value is equal to the D value, thereby performing image processing on all dot pattern lights irradiated onto the pixel array.

[0106] If the m value is different from the D value, the image processing method may add 1 to the m value in step S814 and perform step S803. If the m value is equal to the D value, the image processing method can perform image processing on the next frame in step S815.

[0107] FIG. 9 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. Referring to FIG. 9, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 200 of FIG.

[0108] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), although the scope of the present invention is not limited thereto.

[0109] Computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0110] The processor 1010 is capable of processing data and / or instructions necessary to perform the operations of the image processing device 200 configuration described in FIG.

[0111] The memory 1020 can store data and / or instructions necessary to perform the configuration operations of the image processing device 200 and can be accessed by the processor 1010. For example, the memory 1020 can be implemented as volatile memory (e.g., Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), etc.) or non-volatile memory (e.g., Programmable Read Only Memory (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), flash memory, etc.).

[0112] That is, a computer program for performing the operations of the image processing device 200 disclosed in this document is recorded in the memory 1020 and is executed and processed by the processor 1010, thereby realizing the operations of the image processing device 200.

[0113] The input / output interface 1030 may provide an interface that connects an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010, enabling data to be transmitted and received.

[0114] The communication interface 1040 is configured to be able to send and receive various data to and from an external device (for example, an application processor, an external memory, etc.), and may be a device that can support wired or wireless communication.

[0115] The above description merely exemplifies the technical concept of the present disclosure, and various modifications and variations are possible within the scope of the essential characteristics of the present disclosure, provided that such modifications and variations are made by a person skilled in the art to which the present disclosure pertains. Therefore, the embodiments disclosed in the present disclosure are intended to illustrate, rather than limit, the technical concept of the present disclosure, and the scope of the technical concept of the present disclosure is not limited by such embodiments. The scope of protection of the present disclosure should be interpreted by the scope of the claims below, and all technical concepts within the scope equivalent thereto should be interpreted as being within the scope of the present disclosure.

Claims

1. determining whether to perform averaging processing on at least some of the pixels based on a maximum intensity value among the signal intensity values ​​of pixels included in a pixel group corresponding to a dot pattern; a determination unit that, when it is determined that the averaging process is to be performed, determines, from among the pixels, a target pixel to be subjected to the averaging process based on a result of summing at least some of the intensity values; a calculation unit that calculates a distance value corresponding to the dot pattern by performing an averaging process on the target pixel; 12. An image processing device comprising:

2. The determination unit The image processing device according to claim 1 , wherein it is determined to perform the averaging process if the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value.

3. The determination unit 3. The image processing apparatus of claim 2, wherein the first threshold intensity value is determined using a first total intensity value required to average all of the pixels and a second total intensity value obtained by normalizing and adding together the intensity values ​​of the signals of the pixels.

4. The determination unit The image processing apparatus of claim 2 , wherein the second threshold intensity value is determined to be an intensity value obtained when the distance value is calculated using one pixel of the pixel group.

5. The determination unit The image processing device according to claim 2 , wherein the first threshold intensity value and the second threshold intensity value are determined using dark noise and a signal-to-noise ratio required for a desired distance measurement accuracy.

6. The determination unit The image processing device according to claim 1 , wherein the intensity values ​​are summed one by one in descending order from the maximum intensity value, and the sum calculated in each summing step is compared with a threshold sum to determine the target pixel for the averaging process.

7. The determination unit The image processing apparatus according to claim 6 , wherein a pixel corresponding to the target intensity value summed up until the calculated sum is equal to or greater than the threshold sum is determined as the target pixel.

8. The determination unit The image processing device of claim 7 , wherein if a sum of all the intensity values ​​of the signals of the pixels included in the pixel group is smaller than the threshold sum, all of the pixels included in the pixel group are determined to be target pixels.

9. The determination unit The image processing apparatus according to claim 6 , wherein the threshold sum value is determined to have a different value depending on the number of the object intensity values ​​summed in each summing step.

10. The determination unit The image processing device according to claim 6, wherein the threshold sum value is determined using dark noise and a signal-to-noise ratio required for a desired distance measurement accuracy.

11. The determination unit 2. The image processing apparatus of claim 1, wherein the pixel groups are determined such that the intensity value of a central pixel of the pixel group is the maximum intensity value.

12. The calculation unit The image processing device according to claim 1 , wherein the distance value is calculated by performing the averaging process using the amount of charge generated by the target pixel.

13. a determination unit that generates a summation value for each summation step by summing the signal intensity values ​​of pixels included in a pixel group corresponding to a dot pattern one by one in descending order, and determines a target pixel to be subjected to the averaging process from among the pixels based on the summation value; a calculation unit that calculates a distance value corresponding to the dot pattern by performing an averaging process on the target pixel; 12. An image processing device comprising:

14. The determination unit determining whether to perform the averaging process on at least some of the pixels based on a maximum intensity value among the intensity values; The image processing device according to claim 13 , further comprising: determining the target pixel when determining whether or not to perform the averaging process.

15. The determination unit The image processing device according to claim 14 , wherein it is decided to perform the averaging process if the maximum intensity value is greater than a first threshold intensity value and less than a second threshold intensity value.

16. The determination unit 16. The image processing device of claim 15, wherein the first threshold intensity value is determined using a first total intensity value required to average all of the pixels and a second total intensity value obtained by normalizing and adding together the intensity values ​​of the signals of the pixels.

17. The determination unit The image processing apparatus of claim 15 , wherein the second threshold intensity value is determined to be an intensity value that is obtained when calculating the distance value using one pixel of the pixel group.

18. The determination unit The image processing apparatus of claim 13 , wherein the pixel of interest is determined to be a pixel corresponding to the intensity values ​​of interest that are summed up until the sum is equal to or greater than the threshold sum.

19. The determination unit The image processing device according to claim 18 , wherein if the sum of all the intensity values ​​is less than the threshold sum, all of the pixels included in the pixel group are determined to be target pixels.

20. determining, based on a result of summing at least some of the intensity values ​​of the signals of pixels included in a pixel group corresponding to a dot pattern, a target pixel to be subjected to averaging processing among the pixels; calculating a distance value corresponding to the dot pattern by performing the averaging process using the amount of charge generated by the determined target pixel; An image processing method comprising:

Images