Image processing apparatus, image processing system, and image processing method

By using the data group generator, gradient value manager, and cost space manager in the image processing device, disparity and depth information are generated using the phase and gradient information of pixel values. This solves the problem of insufficient image resolution and depth information accuracy in the prior art and improves the image processing effect.

CN116962865BActive Publication Date: 2026-11-10SK HYNIX INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310251375.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-04-27
Filing Date
2023-03-15
Publication Date
2026-11-10
Estimated Expiration
2043-03-15

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively improve image resolution and the accuracy of depth information when acquiring image information using a single image sensor, especially in image processing using multiple cameras.

Method used

By utilizing the phase and gradient information of pixel values ​​through the data group generator, gradient value manager, and cost space manager in the image processing device, disparity and depth information are generated, thereby achieving efficient image processing.

Benefits of technology

It improves image resolution and the accuracy of depth information, enhancing the effect of image processing, especially in multi-camera image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116962865B_ABST
    Figure CN116962865B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an image processing apparatus, an image processing system and an image processing method. The image processing apparatus comprises a data set generator, a gradient value manager and a cost space manager. The data set generator is configured to receive a pixel value of an image comprising phase information and intensity information, and to determine a data set indicative of a disparity of the image based on a target pixel value, wherein the data set corresponds to a range determined from the phase information. The gradient value manager is configured to determine a gradient value of a region corresponding to the target pixel value and to determine gradient information. The gradient information is determined by applying a threshold value to the gradient value, wherein the threshold value is determined depending on the target pixel value. The cost space manager is configured to determine a cost space by weighted summation of the data sets based on the gradient information.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2022-0052071, filed on April 27, 2022, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The various embodiments disclosed herein generally relate to an image processing system, and more specifically, to an image processing system and image processing method for manipulating pixel information. Background Technology

[0004] Image sensors can be divided into charge-coupled device (CCD) image sensors and complementary metal-oxide-semiconductor (CMOS) image sensors. CMOS image sensors generally have lower manufacturing costs, lower power consumption, and are easier to integrate with peripheral circuits than CCD image sensors.

[0005] Image sensors included in smartphones, tablet PCs, digital cameras, and the like can acquire image information of external objects by converting light reflected from them into electrical signals. Image signal processing devices can perform operations to convert the acquired electrical signals to improve image quality.

[0006] Image signal processing devices can improve image quality by performing image processing operations. Generally, image processing can result in higher resolution when using multiple images from multiple cameras than when using a single image from a single camera. Summary of the Invention

[0007] According to one embodiment, an image processing apparatus may include: a data set generator configured to determine a data set indicating the disparity of an image, wherein pixel values ​​of all pixels included in the image include phase information, and wherein the data set is determined based on a target pixel value among the pixel values ​​corresponding to a range determined according to the phase information; a gradient value manager configured to determine gradient values ​​of a region corresponding to the target pixel value and to determine gradient information by applying a threshold determined according to the target pixel value to the gradient values; and a cost space manager configured to determine a cost space by performing a weighted summation of the data set based on the gradient information.

[0008] According to one embodiment, an image processing system may include an image sensor including microlenses that transmit light received from the outside to pixels, wherein each of the microlenses corresponds to n squared pixels, where n is an integer greater than 2. The image processing system may also include an image processing device configured to obtain depth information of an image based on pixel values ​​received from the image sensor, wherein all of the pixel values ​​include phase information. The image processing device includes: a data set generator configured to determine a data set indicating parallax of the image based on target pixel values ​​among the pixel values ​​that correspond to a range determined according to the phase information; a gradient value manager configured to determine gradient values ​​of a region corresponding to the target pixel values ​​and to determine gradient information by applying a threshold determined according to the target pixel values ​​to the gradient values; and a cost space manager configured to determine a cost space by weighted summation of the data set based on the gradient information and to determine the depth information based on the cost space.

[0009] According to one embodiment, an image processing method may include: receiving pixel values ​​including phase information from all pixels included in an image; determining a data set indicating the disparity of the image based on a target pixel value among the pixel values ​​that corresponds to a range determined according to the phase information; calculating gradient values ​​of a region corresponding to the target pixel value and determining gradient information by applying a threshold determined according to the target pixel value to the gradient value; and determining a cost space by performing a weighted summation on the data set based on the gradient information. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating an exemplary image processing system according to one embodiment.

[0011] Figure 2 This illustrates an embodiment. Figure 1 A diagram of an exemplary image sensor.

[0012] Figure 3 This is a diagram illustrating an exemplary pixel array corresponding to four pixel values ​​and a single microlens according to one embodiment.

[0013] Figure 4 This is a diagram illustrating an exemplary pixel array corresponding to nine pixel values ​​and a single microlens according to one embodiment.

[0014] Figure 5 This is a diagram illustrating an exemplary pixel array corresponding to 16 pixel values ​​and a single microlens according to one embodiment.

[0015] Figure 6This is a diagram illustrating an exemplary method for determining a data group according to one embodiment.

[0016] Figure 7 This is a diagram illustrating an exemplary threshold determined based on pixel values ​​according to one embodiment.

[0017] Figure 8 This is a diagram illustrating an exemplary method of applying a threshold to a gradient value according to one embodiment.

[0018] Figure 9 This is a flowchart illustrating an exemplary method for determining a cost volume according to one embodiment.

[0019] Figure 10 This is a flowchart illustrating an exemplary method for determining a data set according to one embodiment.

[0020] Figure 11 This is a flowchart illustrating an exemplary method for determining gradient information according to one embodiment.

[0021] Figure 12 This is a block diagram illustrating an exemplary electronic device including an image sensor according to one embodiment. Detailed Implementation

[0022] The specific structural or functional descriptions disclosed herein are for illustrative purposes only, describing embodiments based on the concepts of this disclosure. Embodiments based on the concepts of this disclosure may be implemented in various forms and should not be construed as being limited to the specific embodiments set forth herein.

[0023] In the following description, embodiments are illustrated with reference to the accompanying drawings to enable those skilled in the art to implement these embodiments, as well as other embodiments consistent with the technical spirit of this disclosure.

[0024] Various embodiments relate to image processing systems and methods for determining a cost space using pixel values ​​of a single image. The single image can be used for still photographs or for video frames.

[0025] Figure 1 This is a diagram illustrating an example image processing system 10 according to one embodiment.

[0026] refer to Figure 1 The image processing system 10 may include an image sensor 100 and an image processing device 200.

[0027] An image processing system 10 according to one embodiment can acquire an image via, for example, an image sensor 100. The acquired image can be processed by an image processing device 200, which can generate a processed image as an output image. The output image can be output to an external device (not shown). The external device can store and / or display the output image. An image processing system 10 according to one embodiment may include a memory device 240 for storing (containing images) and / or a display (not shown) for displaying images. The memory device 240 may use any one or more volatile and / or non-volatile memory devices as determined according to design and / or implementation specifications. For example, an image processing system 10 according to one embodiment can output an output image to a host in response to a request from a host (not shown).

[0028] Image sensor 100 can generate image data of an object (not shown) whose image is captured using lens 102. Lens 102 may include at least one lens forming an optical system. For example, lens 102 may include microlens 104.

[0029] Image sensor 100 may include a plurality of pixels that receive light. The term "pixel" may be used to refer to a hardware device that detects (or receives) light. The hardware pixel may then output a pixel value of the detected light. Thus, the plurality of pixels in image sensor 100 may generate a plurality of pixel values ​​corresponding to a captured image frame or video frame. The term "pixel" may also be used to refer to a specific location within a frame. Thus, a pixel at a specific location within a frame may be associated with a pixel value provided by a corresponding hardware pixel of image sensor 100. According to one embodiment, all of the plurality of pixel values ​​generated by image sensor 100 may be processed to obtain phase information and brightness (intensity) information of an image.

[0030] Multiple pixel values ​​generated by the image sensor 100 can be sent to the image processing device 200. The image processing device 200 can then obtain the phase information and brightness information of the image by processing the multiple pixel values.

[0031] Image processing apparatus 200 may include a data group generator 210, a gradient value manager 220, a cost space manager 230, and a memory device 240. Image processing apparatus 200 may receive multiple pixel values ​​from image sensor 100. At least some of the multiple pixel values ​​received from image sensor 100 may include phase information and / or brightness information. Therefore, in one embodiment, all of the multiple pixel values ​​received from image sensor 100 may include phase information and brightness information.

[0032] The data set generator 210 can determine a data set indicating the disparity of an image based on a target pixel value among a plurality of pixel values ​​that corresponds to a range determined according to phase information. A "target pixel" can be a specific pixel of the frame of interest. A "target pixel value" can be a pixel value associated with the target pixel. In one embodiment, the disparity of the image can be the binocular disparity of an observer. For example, the term "determine" can indicate using a lookup table, executing software instructions, using hardware circuitry, or performing other functions to obtain the desired output.

[0033] The parallax of an image can refer to the distance between the image sensor 100 and the object. For example, when the distance between the image sensor 100 and the object decreases, the parallax of the image can increase.

[0034] Therefore, the parallax of an image can be inversely proportional to its depth. For example, when the parallax of a portion of an image increases, the depth of that portion of the image can decrease.

[0035] The data group generator 210 can determine a first data group indicating parallax with respect to the horizontal direction of the image based on the target pixel values. The data group generator 210 can also determine a second data group indicating parallax with respect to the vertical direction of the image. According to one embodiment, the number of data groups can vary depending on the parallax of the image.

[0036] The data group generator 210 can determine the average value of the target pixel values ​​in the horizontal direction and the average value of the target pixel values ​​in the vertical direction. The data group generator 210 can determine a first data group based on the average value in the horizontal direction. The data group generator 210 can determine a second data group based on the average value in the vertical direction.

[0037] The data set generator 210 can select a first reference value from the horizontal average values. The data set generator 210 can determine a first data set by comparing the first reference value with the horizontal average value within a predetermined range.

[0038] According to one embodiment, the first data set may indicate the difference between a first reference value and the average values ​​in the remaining horizontal directions (i.e., the average values ​​in the horizontal directions other than the first reference value). Therefore, it can also be said that the first data set may indicate the similarity between pixel values ​​in the horizontal direction.

[0039] The data set generator 210 can select a second reference value from the average values ​​in the vertical direction. The data set generator 210 can determine a second data set by comparing the second reference value with the average value in the vertical direction within a predetermined range. Figure 6 A more detailed description is given of the data group generator 210.

[0040] The gradient value manager 220 can determine the gradient value of the region corresponding to the target pixel value. The gradient value manager 220 can determine gradient information by applying a threshold determined based on the target pixel value to the gradient value. Combined with... Figure 7 and Figure 8 A more detailed description of the gradient value manager 220 is provided.

[0041] For example, gradient value manager 220 can determine a reference value, namely the product of the average gradient value and a threshold determined based on the target pixel value. According to one embodiment, gradient value manager 220 can detect the boundary of a region based on the gradient value of the region corresponding to the target pixel value. Combined with... Figure 7 and Figure 8 A more detailed description of the gradient value manager 220 is provided.

[0042] When the gradient value of a region is less than or equal to the product of the average gradient value and a threshold, the gradient value manager 220 can set the gradient value of the region corresponding to the target pixel value to 0. Similarly, when the gradient value of a region is greater than the product of the average gradient value and a threshold, the gradient value manager 220 can maintain the gradient value of the region corresponding to the target pixel value.

[0043] According to one embodiment, when the gradient value of a region exceeds a threshold, the gradient value manager 220 can detect the region as an image boundary. The gradient value manager 220 can apply a threshold to the gradient values, wherein the threshold varies based on the pixel values.

[0044] People can perceive the same pixel value difference differently depending on the surrounding environment. For example, although the pixel value difference may be the same, people can "see" a larger difference in pixel value in a dark area than in a bright area. Therefore, the embodiment can apply the above-described human visual perception to a threshold. Thus, a larger threshold can be applied to the gradient value in a first region, where the pixel value level is higher than in a second region, where the pixel value level is lower.

[0045] The gradient value manager 220 can determine the gradient values ​​in the horizontal direction and the gradient values ​​in the vertical direction of a region. Therefore, the gradient value manager 220 can determine first gradient information by applying a threshold to the gradient values ​​in the horizontal direction and determine second gradient information by applying a threshold to the gradient values ​​in the vertical direction.

[0046] The cost space manager 230 can determine the cost space by weighted summation of data sets based on gradient information. The cost space manager 230 can also determine the cost space using the average of a first data set and a second data set, where the first data set is weighted based on a first gradient and the second data set is weighted based on a second gradient. Then, the cost space manager 230 can determine depth information based on the cost space. Figure 8 A more detailed description of the cost space manager 230 is provided.

[0047] Figure 2 This illustrates an embodiment. Figure 1 A diagram of an exemplary image sensor.

[0048] refer to Figure 2 For example, an exemplary image sensor may be an image sensor 100, which may include a pixel array 110, a line decoder 120, a timing generator 130, and a signal converter 140.

[0049] According to one embodiment, pixel array 110 may include color filter array 111 and photoelectric conversion layer 113 formed below color filter array 111. Photoelectric conversion layer 113 may include a plurality of photoelectric conversion elements. Corresponding portions of photoelectric conversion elements and color filter array 111 may be referred to as pixels of pixel array 110. Therefore, pixel array 110 may include a plurality of pixels, which may be configured to output pixel signals corresponding to incident light passing through color filter array 111.

[0050] The color filter array 111 may include color filters, each of which allows only light of a specific wavelength (e.g., red, blue, green, etc.) reaching each pixel to pass through. In this disclosure, a color filter may be represented as a color channel. Due to the color filter array 111, the pixel signal from each pixel may include a value corresponding to the intensity of light of a specific wavelength.

[0051] More specifically, each pixel can accumulate photocharge generated by incident light and can generate a pixel signal corresponding to the accumulated photocharge. Each pixel may include: a photoelectric conversion element that converts the accumulated photocharge into an electrical signal; and at least one transistor for processing the electrical signal into a pixel signal. For example, the photoelectric conversion element may be a photodiode, a phototransistor, a photogate, or a pinned photodiode.

[0052] Pixel array 110 may include a plurality of pixels arranged in the row and column directions. Pixel array 110 may generate a plurality of pixel signals (VPXs) per row. Each of the plurality of pixel signals (VPXs) may be an analog pixel signal (VPX).

[0053] According to one embodiment, the microlens 104 can be disposed on the pixel array 110. Although Figure 1 Microlens 104 is shown separated from lens 102, but various embodiments may include microlens 104 as part of lens 102 or separate it from lens 102, depending on design and / or implementation decisions. Each microlens 104 may correspond to an array of n×n pixels, where n is an integer of 2 or greater. An n×n pixel array can be described as “n squared pixels”. Each microlens 104 may physically reside above multiple pixels. While microlens 104 may correspond to n squared pixels, various embodiments do not require this limitation. For example, another embodiment may associate microlens 104 with a rectangular array of m×n pixels (where m and n are both integers greater than 1) or with an array of pixels of another geometry. Furthermore, embodiments may include a first microlens of a first geometry and a second microlens of a second geometry. Thus, in one embodiment, one or more geometries may be present for microlens 104.

[0054] The line decoder 120 can select one row of pixels from multiple rows in the pixel array 110 in response to the control signal and address output by the timing generator 130.

[0055] Signal converter 140 can convert analog pixel signals (VPX) into digital pixel values ​​(DPX). The digital pixel values ​​(DPX) can be output in various patterns. Signal converter 140 can perform correlated double sampling (CDS) on the analog pixel signals output from pixel array 110 in response to control signals from timing generator 130, converting the analog pixel signals into digital pixel values, and outputting the digital pixel values. The digital pixel values ​​can correspond to the intensity of incident light of multiple wavelengths passing through the corresponding color filter array 111.

[0056] Signal converter 140 may include a CDS module and an analog-to-digital converter (ADC) module. The CDS module can sequentially sample and hold a set of reference signals and a set of image signals respectively provided to the multiple column lines included in the pixel array 110. The ADC module can output pixel data obtained by converting the correlated double-sampled signals of each column output from the CDS module into digital signals. For conversion and output, the ADC module may include a comparator and a counter corresponding to each column.

[0057] Furthermore, the image sensor 100 according to one embodiment may also include an output buffer 150. The output buffer 150 may include multiple buffers for storing digital signals output from the signal converter 140. More specifically, the output buffer 150 may latch and output pixel data provided from the signal converter 140 in column units. The output buffer 150 may temporarily store pixel data output from the signal converter 140 and sequentially output pixel data, depending on the control of the timing generator 130. According to one embodiment, the output buffer 150 may be omitted.

[0058] Figure 3 This is a diagram illustrating an exemplary pixel array corresponding to four pixel values ​​and a single microlens according to one embodiment.

[0059] refer to Figure 3 The diagram shows multiple microlenses 310 that transmit light received from outside the image sensor 100 to pixels in the image sensor 100. Each microlens 310 may be located above an array of pixels. Figure 3 An example is shown where four pixels 320 correspond to a single microlens 310. According to... Figure 3 In the illustrated embodiment, four adjacent green pixels 320 may correspond to a single microlens 310. Additionally, four red pixels may correspond to another single microlens 310, and four blue pixels may correspond to another single microlens 310.

[0060] Figure 3 Sixteen pixels are shown corresponding to four microlenses. Multiple pixels 310 corresponding to the same microlens 320 can receive light of the same color. For example, a green pixel 310 may correspond to a single microlens 320. The pixel value of the green pixel 310 may include phase information of the light. For example, the phase may be affected by the position of the pixel 310 and its offset relative to the positions of other pixels 310 in the microlens 320. This can be achieved... Figure 3 As can be seen in the microlens 320, each of the four pixels 310 is positioned differently relative to the other pixels 310.

[0061] Four blue pixels 310 or four red pixels 310 can correspond to another single microlens 320. Similarly, the pixel values ​​of the four blue pixels 310 or four red pixels 310 can also include phase information.

[0062] However, the embodiments disclosed herein are not limited to the embodiment where four pixels correspond to a single microlens. Therefore, the number of pixels corresponding to a single microlens can vary for different embodiments. However, while the number of pixels corresponding to a single microlens can vary, the color of the pixels corresponding to a single microlens can be the same.

[0063] According to one embodiment, the image sensor can generate pixel values ​​that include information about the phase of each pixel. Figure 3 In the illustrated embodiment, the phase information included in the pixel value can be different for each pixel based on the differences in the positions of the four pixels 320 corresponding to the same microlens. For example, this could be because the four pixels 320 corresponding to the same microlens can indicate different phase information based on differences in their positions. According to one embodiment, the depth information of the image can be obtained based on the differences in the phase information included in the four pixel values ​​330 of the microlens 310.

[0064] The boundaries of objects included in an image can be defined based on four pixel values ​​330, which may include phase information and the differences in the positions of the four pixels corresponding to the same microlens. According to one embodiment, the boundaries of objects can be detected by determining gradient values.

[0065] Figure 4 This is a diagram illustrating an exemplary pixel array corresponding to nine pixel values ​​and a single microlens according to one embodiment.

[0066] refer to Figure 4 Nine pixels can correspond to a single microlens. According to... Figure 4 In one embodiment shown, nine adjacent green pixels 410 may correspond to a single microlens. Additionally, nine red pixels may correspond to another single microlens, and nine blue pixels may correspond to yet another single microlens.

[0067] Figure 4 This diagram illustrates 36 pixels corresponding to four microlenses. Multiple pixels corresponding to the same microlens can receive light of the same color. For example, a red pixel can correspond to a single microlens. The pixel value of the red pixel can include phase information of the light received from the microlens reaching the red pixel. This phase information can be influenced by the pixel's position and its offset relative to the positions of other pixels in the microlens. This can be achieved through... Figure 4 As can be seen in the microlens 420, each of the nine pixels is positioned differently relative to the other pixels.

[0068] Nine blue pixels or nine green pixels can correspond to another single microlens. Similarly, the pixel values ​​of the nine blue pixels or nine green pixels can also include phase information.

[0069] According to one embodiment, the image sensor can generate pixel values ​​that include information about the phase in each pixel. Figure 4In the illustrated embodiment, the phase information included in the pixel value can be different for each pixel based on the differences in position of the nine pixels 420 corresponding to the same microlens. For example, this could be because the nine pixels 420 corresponding to the same microlens can indicate different phase information based on differences in position. According to one embodiment, the depth information of the image can be obtained based on the differences in phase information included in the nine pixel values ​​430.

[0070] The boundaries of objects included in an image can be defined based on nine pixel values ​​430, which may include phase information and differences in position of nine pixels 420. According to one embodiment, the boundaries of objects can be detected by determining gradient values.

[0071] Therefore, in one embodiment, an image sensor with a pixel array in which nine pixels correspond to a single microlens can be used. The image sharpness can be improved as the number of pixels corresponding to a single microlens increases.

[0072] Figure 5 This is a diagram illustrating an exemplary pixel array corresponding to 16 pixel values ​​and a single microlens according to one embodiment.

[0073] refer to Figure 5 Sixteen pixels can correspond to a single microlens. According to... Figure 5 In one embodiment shown, 16 adjacent green pixels 510 may correspond to a single microlens. Additionally, 16 red pixels may correspond to another single microlens, and 16 blue pixels may correspond to yet another single microlens.

[0074] Figure 5 This diagram shows 64 pixels corresponding to four microlenses. Pixels corresponding to the same microlens can have the same color. For example, a red pixel can correspond to a single microlens. The pixel value of the red pixel can include phase information of the light received from the microlens reaching the red pixel. This phase information can be influenced by the pixel's position and its offset relative to the positions of other pixels in the microlens. This can be achieved through... Figure 5 As can be seen in the microlens 520, each of the 16 pixels is positioned differently relative to the other pixels.

[0075] Sixteen blue pixels or sixteen green pixels can correspond to another single microlens. Similarly, the pixel values ​​of the sixteen blue pixels or sixteen green pixels can also include phase information.

[0076] According to one embodiment, the image sensor can generate pixel values ​​that include information about the phase in each pixel. Figure 5In one embodiment, the phase information included in the pixel values ​​can differ from each other based on the differences in position of the 16 pixels 520 corresponding to the same microlens. Objects included in the 16 pixels 520 corresponding to the same microlens can indicate different phase information based on differences in position. According to one embodiment, the depth information of the image can be obtained based on the differences in phase information included in the 16 pixel values ​​530.

[0077] The boundaries of objects included in an image can be defined based on 16 pixel values ​​530, which include phase information and differences in position across 16 pixels 520. According to one embodiment, the boundaries of objects can be detected by determining gradient values.

[0078] Figure 3 , Figure 4 and Figure 5 Each of the embodiments shown in the examples is merely illustrative. The number of pixels corresponding to a single microlens is not limited to... Figure 3 , Figure 4 and Figure 5 The example shown.

[0079] Figure 6 This is a diagram illustrating an exemplary method for determining a data group according to one embodiment.

[0080] Figure 6 The pixel values ​​corresponding to a single microlens are shown as an example. For instance, n squared pixels can correspond to microlenses included in an image sensor, where n is an integer of 2 or greater. Figure 6 The illustrated embodiment can determine pixel values ​​in both the horizontal and vertical directions.

[0081] For example, a data group generator such as data group generator 210 can determine data groups in the horizontal direction that indicate the parallax of an image. According to one embodiment, the entries of data included in the data group in the horizontal direction can vary according to the parallax of the image.

[0082] The data group generator can determine the average value (Row1) of the pixel values ​​in the first row of 610. The average value (Row1) of the pixel values ​​in the first row of 610 can be represented by the following formula 1.

[0083] Row1=AVG(X1Y1,X2Y1,…,X N Y1) Formula 1

[0084] The data group generator can determine the average value (Row2) of the pixel values ​​in the second row of 620 pixels. The average value (Row2) of the pixel values ​​in the second row of 620 pixels can be expressed by the following formula 2.

[0085] Row2=AVG(X1Y2,X2Y2,…,X N Y2) Formula 2

[0086] Similarly, the data group generator can determine the average value (RowN) of the pixel values ​​in row N, 630. The average value (RowN) of the pixel values ​​in row N, 630 can be represented by the following formula 3.

[0087] RowN = AVG(X1Y) N X2Y N , ..., X N Y N )Formula 3

[0088] The data set generator can select a first reference value from the horizontal average values ​​(Row1, Row2, ..., RowN). The data set generator can generate a horizontal data set comprising data determined by shifting the horizontal average values ​​(Row1, Row2, ..., RowN) and comparing them to the first reference value within a predetermined range.

[0089] For example, the data group generator can select the average pixel value (Row1) of the first row 610 from the horizontal average values ​​(Row1, Row2, ..., RowN) as a first reference value. The data group generator can determine data (DataRow1) by shifting the average pixel value (Row2) of the second row 620 and comparing it with the first reference value. Alternatively, the data group generator can select the average pixel value (Row2) of the second row 620 as the first reference value and determine data (DataRow2) by shifting the average pixel value (Row3) of the third row and comparing it with the first reference value. Similarly, the data group generator can select the average pixel value (RowN-1) of the (N-1)th row as the first reference value and determine data (DataRowN-1) by shifting the average pixel value (RowN) of the Nth row 630 and comparing it with the first reference value. The data group generator can determine a first data group, which includes multiple data lines (DataRow1, DataRow2, ..., DataRowN-1) determined by comparing the average pixel value with a first reference value.

[0090] The data group generator can determine the average value (Col1) of the pixel values ​​in the first column of 640. The average value (Col1) of the pixel values ​​in the first column of 640 can be represented by the following formula 4.

[0091] Col1=AVG(X1Y1,X1Y2,…,X1Y N)Formula 4

[0092] The data group generator can determine the average value (Col2) of the pixel values ​​in the second column of 650. The average value (Col2) of the pixel values ​​in the second column of 650 can be represented by the following formula 5.

[0093] Col2=AVG(X2Y1,X2Y2,…,X2Y N )Formula 5

[0094] Similarly, the data group generator can determine the average value (ColN) of the pixel values ​​in column N, 660. The average value (ColN) of the pixel values ​​in column N, 660 can be represented by the following formula 6.

[0095] ColN = AVG(X) N Y1, X N Y2, ..., X N Y N )Formula 6

[0096] The data set generator can select a second reference value from the vertical average values ​​(Col1, Col2, ..., ColN). The data set generator can determine a vertical data set comprising data generated by shifting the vertical average values ​​(Col1, Col2, ..., ColN) and comparing them with a second reference value within a predetermined range.

[0097] For example, the data group generator can select the average value (Col1) of the pixel values ​​in the first column (640 pixels) from the vertical average values ​​(Col1, Col2, ..., ColN) as a second reference value. The data group generator can determine data (DataCol1) by shifting the average value (Col2) of the pixel values ​​in the second column (650 pixels) and comparing it with the second reference value. The data group generator can select the average value (Col2) of the pixel values ​​in the second column (650 pixels) as the second reference value, and determine data (DataCol2) by shifting the average value (Col3) of the pixel values ​​in the third column and comparing it with the second reference value. Similarly, the data group generator can select the average value (ColN-1) of the pixel values ​​in the (N-1)th column as the second reference value, and determine data (DataColN-1) by shifting the average value (ColN) of the pixel values ​​in the Nth column (660 pixels) and comparing it with the second reference value. The data group generator can determine a second data group, which includes multiple data points (DataCol1, DataCol2, ..., DataColN-1) determined by comparing the average pixel value with a second reference value.

[0098] Figure 7 This is a diagram illustrating an exemplary threshold determined based on pixel values ​​according to one embodiment.

[0099] refer to Figure 7 The threshold applied to the gradient value can vary depending on the pixel value. As previously mentioned, depending on the level of the pixel value, people perceive the same difference in pixel values ​​as different. According to one embodiment, the aforementioned human visual characteristics can be applied to the threshold. Figure 7 The image shows pixel values ​​and thresholds according to one embodiment. However, embodiments of this disclosure are not limited to those described herein. Figure 7 The example shown.

[0100] exist Figure 7 In this context, the x-axis can represent the intensity of the grayscale image. According to one embodiment, the image intensity can refer to the level of pixel values. Figure 7 In this context, the y-axis can be a varying threshold. For example, the threshold can vary based on the level of pixel values. Therefore, Figure 7 Different thresholds depending on the pixel value can be shown.

[0101] For the same pixel value difference, one might perceive a larger difference in pixel values ​​in dark areas than in bright areas. Therefore, the threshold applied to the gradient value in areas with larger pixel values ​​can be greater than the threshold applied to the gradient value in areas with smaller pixel values.

[0102] exist Figure 7 In the range where pixel values ​​are less than 16, the threshold can be 40. In the range where pixel values ​​are greater than or equal to 16 and less than 32, the threshold can increase proportionally to the pixel value. The threshold corresponding to dark areas with pixel values ​​less than or equal to 16 can be less than the threshold corresponding to bright areas with pixel values ​​greater than or equal to 64. In dark areas, even when the difference in pixel values ​​is very small, one pixel can be distinguished from another.

[0103] According to one embodiment, a gradient value manager, such as gradient value manager 220, can determine a threshold to apply to gradient values. Therefore, the threshold can be different for the same pixel value difference in different regions. The gradient value manager can determine that the threshold applied to gradient values ​​in dark areas is less than the threshold applied to gradient values ​​in bright areas. For example, the gradient value manager can determine the threshold by using a lookup table, executing software and / or hardware instructions to compute the threshold, or performing other functions to determine the threshold.

[0104] Figure 8 This is a diagram illustrating an exemplary method of applying a threshold to a gradient value according to one embodiment.

[0105] refer to Figure 8The gradient value of the region corresponding to the target pixel within a predetermined range can be determined as 0. Regions with a gradient value of 0 can be excluded from detection as boundaries. Figure 8 In this context, the range within which the gradient value is set to 0 can vary depending on the pixel value.

[0106] exist Figure 8 In this diagram, the x-axis represents the gradient value, and the y-axis represents the output gradient value. The range of output gradient values ​​of 0 can vary depending on the pixel value.

[0107] A gradient value manager, similar to gradient value manager 220, can determine a reference value, which is the product of the average gradient values ​​and a threshold determined based on the target pixel value. According to one embodiment, the reference value can vary based on the target pixel value.

[0108] The gradient value manager compares the gradient value of a corresponding region with a reference value. When the gradient value of the corresponding region is greater than the reference value, the gradient value manager outputs the unchanged gradient value of the corresponding region. When the gradient value of the corresponding region is less than or equal to the reference value, the gradient value manager outputs 0 as the gradient value of the corresponding region.

[0109] According to one embodiment, a gradient value manager can determine the gradient values ​​in the horizontal direction and the gradient values ​​in the vertical direction of a region corresponding to a target pixel. The gradient value manager can determine first gradient information by applying a threshold to the gradient values ​​in the horizontal direction and determine second gradient information by applying a threshold to the gradient values ​​in the vertical direction.

[0110] Return to reference Figure 6 , Figure 6 The method for the gradient value manager to determine the first gradient information and the second gradient information is shown. The gradient value manager can determine the gradient value (GradientRow1) of the first row based on the pixel values ​​of the first row 610. The gradient value manager can determine the adaptive gradient value (AGTRow1) of the first row by applying a threshold determined based on the pixel values ​​of the first row 610 to the gradient value (GradientRow1) of the first row.

[0111] The gradient value manager can determine the gradient value (GradientRow2) of the second row based on the pixel values ​​of the second row (620). The gradient value manager can determine the adaptive gradient value (AGTRow2) of the second row by applying the gradient value (GradientRow2) of the second row to a threshold determined based on the pixel values ​​of the second row (620).

[0112] Similarly, the gradient value manager can determine the adaptive gradient value (AGTRowN-1) of the (N-1)th row based on the pixel value of the (N-1)th row. The gradient value manager can determine the first gradient information including the adaptive gradient values ​​(AGTRow1, AGTRow2, ..., AGTRowN-1).

[0113] The gradient value manager can determine the gradient value (GradientCol1) of the first column based on the pixel values ​​of the first column 640. The gradient value manager can determine the adaptive gradient value (AGTCol1) of the first column by applying the gradient value (GradientCol1) of the first column to a threshold determined based on the pixel values ​​of the first column 640.

[0114] The gradient value manager can determine the gradient value (GradientCol2) of the second column based on the pixel value of the second column 650. The gradient value manager can determine the adaptive gradient value (AGTCol2) of the second column by applying the gradient value (GradientCol2) of the second column to a threshold determined based on the pixel value of the second column 650.

[0115] Similarly, the gradient value manager can determine the adaptive gradient value (AGTColN-1) of the (N-1)th column based on the pixel value of the (N-1)th column. The gradient value manager can determine second gradient information including the adaptive gradient values ​​(AGTCol1, AGTCol2, ..., AGTColN-1).

[0116] According to one embodiment, for example, a cost space manager such as cost space manager 230 can determine the cost space of an image using the average of a first data set weighted based on first gradient information and a second data set weighted based on second gradient information. The cost space manager can obtain depth information of the image based on the cost space of the image.

[0117] For example, the cost space manager can determine the cost space (costvolumRow) in the horizontal direction. The cost space (costvolumRow) in the horizontal direction can be obtained by dividing the weighted sum of the adaptive gradient values ​​(AGTRow1, AGTRow2, ..., AGTRowN-1) included in the first gradient information and the data (DataRow1, DataRow2, ..., DataRowN-1) included in the first data set by the sum of the adaptive gradient values ​​(AGTRow1, AGTRow2, ..., AGTRowN-1) included in the first gradient information. The cost space (costvolumRow) in the horizontal direction can be represented by the following Equation 7.

[0118] costvolumRow=(abs(AGTRow1)*DataRow1+abs(AGTRow2)*DataRow2+…+abs(AGTRowN-1)*DataRowN-1)) / (abs(AGTRow1)+abs(AGTRow2)+…+abs(AGTRowN-1)) Formula 7

[0119] The cost space manager can determine the cost space (costvolumCol) in the vertical direction. The cost space (costvolumCol) in the vertical direction can be obtained by dividing the weighted sum of the adaptive gradient values ​​(AGTCol1, AGTCol2, ..., AGTColN-1) included in the second gradient information and the data (DataCol1, DataCol2, ..., DataColN-1) included in the second data set by the sum of the adaptive gradient values ​​(AGTCol1, AGTCol2, ..., AGTColN-1) included in the second gradient information. The cost space (costvolumCol) in the vertical direction can be represented by the following Equation 8.

[0120] costvolumCol=(abs(AGTCol1)*DataCol1+abs(AGTCol2)*DataCol2+…+abs(AGTColN-1)*DataColN-1)) / (abs(AGTCol1)+abs(AGTCol2)+…+abs(AGTColN-1)) Formula 8

[0121] The cost space manager can determine the cost space of an image. The cost space of an image can be the average of the cost space (cost spaceRow) in the horizontal direction and the cost space (cost spaceCol) in the vertical direction. The cost space of an image can be represented by the following formula 9.

[0122] costvolum=AVG(costvolumRow, costvolumCol) formula 9

[0123] Figure 9 This is a flowchart illustrating an exemplary method for determining a cost space according to one embodiment.

[0124] refer to Figure 9For example, an image processing apparatus such as image processing device 200 can determine a cost space that can be used to obtain depth information of an image based on pixel values ​​received, for example, from image sensor 100 or from a device other than image processing apparatus 200. For example, pixel values ​​can be received from a device outside of image processing system 10 (not shown). The image processing apparatus can determine a data set and gradient information for each in the horizontal and vertical directions, and then determine the cost space of the image by weighted summation of the determined data set and the determined gradient information.

[0125] In step S910, the image processing device can receive pixel values ​​from all pixels included in the image. The pixel values ​​may include one or both of phase information and brightness information. The image processing device can obtain depth information of the image based on the received pixel values.

[0126] In step S920, the data group generator of the image processing apparatus can determine a data group indicating the parallax of the image. This can be done based on the target pixel value among the pixel values ​​that corresponds to the range determined according to the phase information.

[0127] The data set generator can determine a first data set indicating parallax in the horizontal direction of the image based on the target pixel values. Furthermore, the data set generator can determine a second data set indicating parallax in the vertical direction of the image based on the target pixel values.

[0128] The data set generator can determine data sets in the horizontal and vertical directions that indicate the parallax of an image. The number of data entries included in each data set can vary according to the parallax of the image.

[0129] The data set generator can select a first reference value from the horizontal average values ​​(Row1, Row2, ..., RowN). The data set generator can determine a horizontal data set comprising data obtained by shifting the horizontal average values ​​(Row1, Row2, ..., RowN) and comparing them with the first reference value within a predetermined range.

[0130] The data set generator can select a second reference value from the vertical average values ​​(Col1, Col2, ..., ColN). The data set generator can determine a vertical data set comprising data obtained by shifting the vertical average values ​​(Col1, Col2, ..., ColN) and comparing them with a second reference value within a predetermined range.

[0131] In step S930, the gradient value manager of the image processing device can determine the gradient value and threshold of the region corresponding to the target pixel value. Then, the gradient value manager can determine the gradient information by applying the threshold to the gradient value.

[0132] According to one embodiment, the gradient value manager can determine a threshold to apply to gradient values, the threshold varying based on the pixel values ​​of the corresponding region. For example, the gradient value manager can determine that the threshold applied to gradient values ​​in dark areas is less than the threshold applied to gradient values ​​in bright areas.

[0133] The gradient value manager can determine a reference value, which is the product of the average gradient values ​​and a threshold determined based on the target pixel value. According to one embodiment, the reference value can vary based on the target pixel value.

[0134] The gradient value manager compares the gradient value of a corresponding region with a reference value. When the gradient value of the corresponding region is greater than the reference value, the gradient value manager outputs the unchanged gradient value of the corresponding region. When the gradient value of the corresponding region is less than or equal to the reference value, the gradient value manager determines that the gradient value of the corresponding region is 0.

[0135] According to one embodiment, a gradient value manager can determine the gradient values ​​in the horizontal direction and the gradient values ​​in the vertical direction of a region corresponding to a target pixel. The gradient value manager can determine first gradient information by applying a threshold to the gradient values ​​in the horizontal direction and determine second gradient information by applying a threshold to the gradient values ​​in the vertical direction.

[0136] At step S940, the cost space manager of the image processing device can determine the cost space by performing a weighted summation of the data set based on gradient information.

[0137] According to one embodiment, the cost space manager can determine the cost space of an image using the average of a first data set weighted based on first gradient information and a second data set weighted based on second gradient information. The cost space manager can then obtain the depth information of the image based on its cost space.

[0138] Figure 10 This is a flowchart illustrating an exemplary method for determining a data group according to one embodiment.

[0139] refer to Figure 10 For example, the data group generator of an image processing apparatus such as image processing apparatus 200 can determine data groups in the horizontal direction and data groups in the vertical direction that indicate the parallax of an image. The data group generator can determine different data based on the parallax of the image.

[0140] At step S1010, the data group generator can determine an average value with respect to a predetermined direction based on the pixel values. According to one embodiment, the predetermined direction can be a horizontal direction or a vertical direction.

[0141] The data set generator can determine the average value (Row1, Row2, ..., RowN) in the horizontal direction. Furthermore, the data set generator can determine the average value (Col1, Col2, ..., ColN) in the vertical direction.

[0142] At step S1020, the data set generator can select a reference value from the determined average values. The data set generator can select a first reference value from the horizontal average values ​​(Row1, Row2, ..., RowN) and a second reference value from the vertical average values ​​(Col1, Col2, ..., ColN).

[0143] For example, the data group generator can select the average pixel value (Row1) of the first row from the horizontal average values ​​(Row1, Row2, ..., RowN) as the first reference value. The data group generator can select the average pixel value (Col1) of the first column from the vertical average values ​​(Col1, Col2, ..., ColN) as the second reference value.

[0144] At step S1030, the data group generator can determine a data group comprising data obtained by comparing reference values ​​with average values ​​within a predetermined range. The data group generator can determine a data group in the horizontal direction, comprising data obtained by shifting the average values ​​(Row1, Row2, ..., RowN) in the horizontal direction and comparing them with a first reference value within the predetermined range. The data group generator can determine a data group in the vertical direction, comprising data obtained by shifting the average values ​​(Col1, Col2, ..., ColN) in the vertical direction and comparing them with a second reference value within the predetermined range.

[0145] For example, a data group generator can select the average pixel value of the first row (Row1) as a first reference value, and determine data (DataRow1) by shifting the average pixel value of the second row (Row2) and comparing it with the first reference value. Similarly, a data group generator can select the average pixel value of the (N-1)th row (RowN-1) as a first reference value, and determine data (DataRowN-1) by shifting the average pixel value of the Nth row (RowN) and comparing it with the first reference value. The data group generator can determine a first data group, which includes multiple data entries (DataRow1, DataRow2, ..., DataRowN-1) determined by comparing the average value with the first reference value.

[0146] According to one embodiment, the data group generator can select the average value (Col1) of the pixel values ​​in the first column as a second reference value, and determine data (DataCol1) by shifting the average value (Col2) of the pixel values ​​in the second column and comparing it with the second reference value. Similarly, the data group generator can select the average value (ColN-1) of the pixel values ​​in the (N-1)th column as a second reference value, and determine data (DataColN-1) by shifting the average value (ColN) of the pixel values ​​in the Nth column and comparing it with the second reference value. The data group generator can determine a second data group, which includes multiple data entries (DataCol1, DataCol2, ..., DataColN-1) determined by comparing the average value with the second reference value.

[0147] Figure 11 This is a flowchart illustrating an exemplary method for determining gradient information according to one embodiment.

[0148] refer to Figure 11 For example, the gradient value manager of an image processing apparatus such as image processing apparatus 200 can determine the gradient value of a region corresponding to a target pixel value, and determine gradient information by applying a threshold determined based on the target pixel value to the gradient value. The gradient value manager can determine first gradient information by applying a threshold to the gradient value in the horizontal direction and determine second gradient information by applying a threshold to the gradient value in the vertical direction.

[0149] In step S1110, the gradient value manager can determine the gradient value in the horizontal direction based on the pixel values. The gradient value manager can determine the gradient value (GradientRow1) of the first row based on the pixel values ​​of the first row. The gradient value manager can determine the gradient value (GradientRowN-1) of the (N-1)th row based on the pixel values ​​of the (N-1)th row.

[0150] At step S1120, the gradient value manager can determine the gradient value in the vertical direction based on the pixel values. The gradient value manager can determine the gradient value (GradientCol1) of the first column based on the pixel values ​​of the first column. The gradient value manager can determine the gradient value (GradientColN-1) of the (N-1)th column based on the pixel values ​​of the (N-1)th column.

[0151] At step S1130, the gradient value manager can determine first gradient information by applying a threshold to the gradient values ​​in the horizontal direction and second gradient information by applying a threshold to the gradient values ​​in the vertical direction. According to one embodiment, the gradient value manager can determine the threshold to be applied to the gradient values, the threshold varying depending on the pixel values ​​of the corresponding region. The gradient value manager can determine that the threshold applied to gradient values ​​in dark areas is smaller than the threshold applied to gradient values ​​in bright areas.

[0152] The gradient value manager can determine a reference value, which is the product of the average gradient values ​​and a threshold determined based on the target pixel value. According to one embodiment, the reference value can vary based on the target pixel value.

[0153] The gradient value manager compares the gradient value of a corresponding region with a reference value. When the gradient value of the corresponding region is greater than the reference value, the gradient value manager outputs the unchanged gradient value of the corresponding region. When the gradient value of the corresponding region is less than or equal to the reference value, the gradient value manager determines that the gradient value of the corresponding region is 0.

[0154] According to one embodiment, a gradient value manager can determine the gradient values ​​in the horizontal direction and the gradient values ​​in the vertical direction of a region corresponding to a target pixel. The gradient value manager can determine first gradient information by applying a threshold to the gradient values ​​in the horizontal direction and determine second gradient information by applying a threshold to the gradient values ​​in the vertical direction.

[0155] The gradient value manager can determine the adaptive gradient value (AGTRow1) of the first row by applying the gradient value (GradientRow1) of the first row to a threshold determined based on the pixel values ​​of the first row. Similarly, the gradient value manager can determine the first gradient information, which includes the adaptive gradient values ​​(AGTRow1, AGTRow2, ..., AGTRowN-1).

[0156] The gradient value manager can determine the adaptive gradient value (AGTCol1) of the first column by applying the gradient value (GradientCol1) of the first column to a threshold determined based on the pixel values ​​of the first column. Similarly, the gradient value manager can determine second gradient information including adaptive gradient values ​​(AGTCol1, AGTCol2, ..., AGTColN-1).

[0157] Figure 12 This is a block diagram illustrating an exemplary electronic device including an image sensor according to one embodiment.

[0158] refer to Figure 12 The electronic device 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input device 2050, and an output device 2060. Although Figure 12 Not shown, but electronic device 2000 may also include a port capable of communicating with a video card, sound card, memory card, universal serial bus (USB) device and / or other electronic devices.

[0159] Image sensor 2010 can determine image data corresponding to incident light. The image data can be transmitted to processor 2020 and processed thereon. Output device 2060 can display the image data. Storage device 2030 can store the image data. Processor 2020 can control the operation of image sensor 2010, output device 2060, and / or storage device 2030.

[0160] The processor 2020 may be an image processing device that performs operations to process pixel data received from the image sensor 2010 and outputs processed image data. For example, the processing performed by the processor 2020 may include electronic image stabilization (EIS), interpolation, tone correction, image quality correction, resizing, etc.

[0161] The processor 2020 can be configured as a separate chip from the image sensor 2010. For example, the processor 2020 can be formed in a multi-chip package. According to another embodiment, the processor 2020 can be configured as a single chip that is part of the image sensor 2010.

[0162] Processor 2020 can perform and control the operation of electronic device 2000. According to one embodiment, processor 2020 can be a microprocessor, central processing unit (CPU), application processor (AP), microcontroller, or any suitable device capable of executing instructions or command sets. Processor 2020 can be coupled to and communicate with storage device 2030, memory device 2040, input device 2050, and output device 2060 via address bus, control bus, and / or data bus.

[0163] According to one embodiment, processor 2020 can receive pixel values ​​including phase information and brightness information of an image, and a data set indicating the disparity of the image based on a target pixel value corresponding to a range determined according to the phase information. Processor 2020 can determine gradient values ​​of a region corresponding to the target pixel value and determine gradient information by applying a threshold determined according to the target pixel value to the gradient values. Processor 2020 can determine a cost space by weighted summation of the data set based on the gradient information. Depth information can be determined using the cost space. Therefore, processor 2020 can perform at least some functions of image processing apparatus 200.

[0164] For example, storage device 2030 may include one or more non-volatile storage devices, such as flash memory devices, solid-state drives (SSDs), hard disk drives (HDDs), optical disc read-only memories (CD-ROMs), etc.

[0165] Memory device 2040 may store data for performing operations of electronic device 2000. For example, memory device 2040 may include one or more volatile memory devices such as dynamic random access memory (DRAM) devices and static random access memory (SRAM) devices; and one or more non-volatile memory devices such as erasable programmable read-only memory (EPROM) devices, electrically erasable programmable read-only memory (EEPROM) devices, flash memory devices, etc. Processor 2020 may execute a set of commands stored in memory device 2040 to control image sensor 2010 and output device 2060, as well as other devices designed or implemented.

[0166] For example, the input device 2050 may include input tools such as a keyboard, keypad, mouse, stylus, etc.

[0167] The image sensor 2010 can be implemented in a variety of packages. For example, at least some configurations of the image sensor 2010 can use packages such as, for example, package-on-package (PoP), ball grid array (BGA), chip-scale package (CSP), plastic leaded chip carrier (PLCC), plastic dual in-line package (PDIP), die in waffle package form, die in wafer form, chip-on-board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system package (SIP), multi-chip package (MCP), wafer-level fabrication package (WFP), wafer-level processing stack package (WSP), etc.

[0168] Electronic device 2000 can be interpreted as any computing system using image sensor 2010. Electronic device 2000 can be provided in the form of packaged modules, components, etc. For example, electronic device 2000 can be a digital camera, mobile device, smartphone, personal computer (PC), tablet PC, laptop computer, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, autonomous vehicle, etc.

[0169] The scope of this disclosure is described by the claims, not by the detailed descriptions set forth above. The scope of this disclosure should be interpreted as including not only the scope of the claims but also all changes or modifications derived from the meaning, scope, and equivalents of the claims.

[0170] According to various embodiments of the present disclosure, an image processing system capable of obtaining depth information using pixel values ​​of a single image can be provided.

Claims

1. An image processing apparatus, comprising: Data set generator, which: Receives pixel values, including phase information and brightness information of the image, from a device external to the image processing device; and A data group is determined based on a target pixel value, wherein the target pixel value corresponds to a range determined according to the phase information; Gradient value manager, which: Determine the gradient value of the region corresponding to the target pixel value; and Gradient information is determined by applying a threshold to the gradient value, wherein the threshold is determined based on the target pixel value; and A cost space manager that determines the cost space by performing a weighted summation of the data set based on the gradient information.

2. The image processing apparatus according to claim 1, wherein, Based on the target pixel value, the data group generator: Determine a first set of data regarding the horizontal direction of the image; and Determine a second set of data regarding the vertical direction of the image.

3. The image processing apparatus according to claim 2, wherein, The data set generator: Determine the average value of the target pixel in the horizontal direction and the average value of the target pixel in the vertical direction; The first data group is determined based on the horizontal average value of the target pixel values; and The second data group is determined based on the average value of the target pixel value in the vertical direction.

4. The image processing apparatus according to claim 3, wherein, The data set generator: Select a first reference value from the horizontal average of the target pixel values; and The first data set is determined by comparing the first reference value with the horizontal average of the target pixel values ​​within a predetermined range.

5. The image processing apparatus according to claim 3, wherein, The data set generator: Select a second reference value from the average value of the target pixel value in the vertical direction; and The second data set is determined by comparing the second reference value with the average value of the target pixel value in the vertical direction within a predetermined range.

6. The image processing apparatus according to claim 2, wherein, The gradient value manager: Determine the gradient values ​​in the horizontal and vertical directions of the region; First gradient information is generated by applying the threshold to the gradient value in the horizontal direction; and The second gradient information is determined by applying the threshold to the gradient value in the vertical direction.

7. The image processing apparatus according to claim 6, wherein, The cost space manager uses the average of the following items to determine the cost space: The first data group, weighted based on the first gradient information, and The second data group is weighted based on the second gradient information.

8. The image processing apparatus according to claim 1, wherein, When the gradient value in the region is less than or equal to the product of the average gradient value and the threshold, the gradient value manager determines the gradient value of the region to be 0.

9. The image processing apparatus according to claim 1, wherein, When the gradient value in the region is greater than the product of the average gradient value and the threshold, the gradient value manager keeps the gradient value in the region unchanged.

10. An image processing system, comprising: An image sensor that: determines pixel values, including phase and brightness information of an image, based on light received from the outside; and An image processing apparatus that: obtains depth information of an image based on the pixel values ​​received from the image sensor. The image sensor includes microlenses, each of which corresponds to n squared pixels, where n is an integer of 2 or greater. The image processing device includes: A data group generator that determines a data group based on a target pixel value among the pixel values, the target pixel value corresponding to a range determined according to the phase information; A gradient value manager, which: determines the gradient value of a region corresponding to the target pixel value and determines gradient information by applying a threshold determined based on the target pixel value to the gradient value; and A cost space manager that determines a cost space by weighted summation of the data set based on the gradient information and determines the depth information based on the cost space.

11. The image processing system according to claim 10, wherein, The data group generator: determines the average value of the target pixel value in the horizontal direction and the average value of the target pixel value in the vertical direction, determines a first data group in the horizontal direction of the image based on the average value of the target pixel value in the horizontal direction, and determines a second data group in the vertical direction of the image based on the average value of the target pixel value in the vertical direction.

12. The image processing system according to claim 11, wherein, The data group generator determines the first data group by comparing a first reference value selected from the horizontal average values ​​of the target pixel values ​​with the horizontal average values ​​of the target pixel values ​​within a predetermined range, and determines the second data group by comparing a second reference value selected from the vertical average values ​​of the target pixel values ​​with the vertical average values ​​of the target pixel values ​​within the predetermined range.

13. The image processing system according to claim 11, wherein, The gradient value manager: determines the gradient value in the horizontal direction and the gradient value in the vertical direction of the region corresponding to the target pixel value, and determines first gradient information by applying the threshold to the gradient value in the horizontal direction and determines second gradient information by applying the threshold to the gradient value in the vertical direction.

14. The image processing system according to claim 13, wherein, The cost space manager determines the cost space using the average of the first data set and the second data set, wherein the first data set is weighted based on the first gradient information and the second data set is weighted based on the second gradient information.

15. An image processing method, comprising: Receive pixel values ​​from the outside, including brightness and phase information of the image; A data group is determined based on the target pixel value corresponding to the range determined according to the phase information; Calculate the gradient value of the region corresponding to the target pixel value and determine gradient information by applying a threshold determined based on the target pixel value to the gradient value; as well as The cost space is determined by weighted summation of the data set based on the gradient information.

16. The image processing method according to claim 15, wherein, Determining the data set includes: Based on the target pixel values, a first data set is determined regarding the horizontal direction of the image; and A second data set is determined based on the target pixel value in the vertical direction of the image.

17. The image processing method according to claim 16, wherein, Generating the first data group includes: Determine the average value of the target pixel value in the horizontal direction; Select a first reference value from the horizontal average of the target pixel values; and The first data set is determined by comparing the first reference value with the horizontal average of the target pixel values ​​within a predetermined range.

18. The image processing method according to claim 16, wherein, Determining the second data group includes: Calculate the average value of the target pixel value in the vertical direction; Select a second reference value from the average value of the target pixel value in the vertical direction; and The second data set is determined by comparing the second reference value with the average value of the target pixel value in the vertical direction within a predetermined range.

19. The image processing method according to claim 16, wherein, Determining the gradient information includes: Determine the horizontal and vertical gradient values ​​of the region corresponding to the target pixel value; and A first gradient information is determined by applying the threshold to the gradient value in the horizontal direction, and a second gradient information is determined by applying the threshold to the gradient value in the vertical direction.

20. The image processing method according to claim 19, further comprising: The depth information of the image is determined based on the cost space. Determining the cost space includes: calculating the cost space using the average of the first data group weighted based on the first gradient information and the second data group weighted based on the second gradient information.

Citation Information

Patent Citations

  • Manufacturing method of interior for vehicle

    KR1020220052071A

  • Cost function generation method and code function generation device

    CN107403448A

  • Depth map acquisition method, device, electronic equipment and computer readable storage medium

    CN112866674A