Image processing apparatus and image processing method

Abstract

To provide an image processing device capable of properly correcting depth information in accordance with the color of a subject.SOLUTION: An image processing device 100 comprises: a detection unit 110 which detects a subject area from a photographic image; an acquisition unit 120 which acquires color information and depth information included in the subject area for each pixel; a color determination unit 130 which determines the color of a pixel based on the color information; and a correction unit 140 which corrects the depth information of the pixel based on a result of the determination by the color determination unit. The correction unit 140 corrects the depth information of a pixel based on the depth information of a chromatic color pixel located near the pixel when the pixel is a black pixel.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus and an image processing method. [Background technology]

[0002] One known technique for measuring the distance (depth) from an imaging device to a subject is the Time of Flight (ToF) method. A ToF sensor using the ToF method illuminates the subject with infrared light for distance measurement, and receives the reflected light with an infrared image sensor. The ToF sensor can calculate the distance between the subject and the imaging device by detecting the time difference between illumination and reception for each pixel.

[0003] As a related technology, for example, Patent Document 1 discloses an image processing device including a first type of sensor, a second type of sensor, and a control circuit. In this image processing device, the control circuit receives an input color image frame from the first type of sensor and an input depth image corresponding to the input color image frame from the second type of sensor. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Special Publication No. 2021-521543 [Overview of the project] [Problems that the invention aims to solve]

[0005] The accuracy of the distance (depth value) obtained by a ToF sensor varies depending on the color of the subject. This is because the reflectivity differs depending on the color of the subject. Therefore, even if subjects of different colors are located at the same distance from the ToF sensor, the measured distance may differ depending on the color of the subject.

[0006] Furthermore, because black objects have low reflectivity, ToF sensors may not be able to properly acquire depth values. As a result, when a 3D image is generated based on depth values, depth values ​​for black objects will not exist, leading to problems such as parts that should be flat appearing as voids, or flat objects being rendered as bumpy objects in 3D.

[0007] The purpose of this disclosure is to provide an image processing apparatus and an image processing method that can appropriately correct depth information according to the color of the subject, in view of the above-mentioned problems. [Means for solving the problem]

[0008] The image processing apparatus relating to this disclosure is A detection unit that detects the subject area from the captured image, An acquisition unit that acquires color information and depth information for each pixel included in the subject area, A color determination unit that determines the color of the pixel based on the aforementioned color information, The system includes a correction unit that corrects the depth information of the pixels based on the determination result of the color determination unit, If the pixel is a black pixel, the correction unit corrects the depth information of the pixel based on the depth information of a chromatic pixel in the vicinity of the pixel.

[0009] The image processing method relating to this disclosure is: A detection step to detect the subject area from the captured image, An acquisition step of acquiring color information and depth information for each pixel included in the subject area, A color determination step in which the color of the pixel is determined based on the color information, The correction step includes correcting the depth information of the pixel based on the determination result in the color determination step, In the correction step, if the pixel is a black pixel, the depth information of the pixel is corrected based on the depth information of a chromatic pixel in the vicinity of the pixel. [Effects of the Invention]

[0010] According to the image processing apparatus and the image processing method according to the present disclosure, depth information can be appropriately corrected according to the color of the subject.

Brief Description of the Drawings

[0011] [Figure 1] It is a block diagram showing the configuration of the imaging system according to the embodiment. [Figure 2] As an example of a captured image, it is a diagram showing a captured image including a plurality of subjects. [Figure 3] It is a diagram showing an example of the arrangement of RGB values and depth values in the subject area. [Figure 4] It is an explanatory diagram of the first correction method for black pixels. [Figure 5] It is a diagram for explaining a modification example of the first correction method for black pixels. [Figure 6] It is an explanatory diagram of the second correction method for black pixels. [Figure 7] It is an explanatory diagram of the third correction method for black pixels. [Figure 8] It is an explanatory diagram of the third correction method for black pixels. [Figure 9] It is a diagram for explaining a modification example of the third correction method for black pixels. [Figure 10] It is an explanatory diagram of the fourth correction method for black pixels. [Figure 11] It is an explanatory diagram of the fourth correction method for black pixels. [Figure 12] It is a flowchart showing the processing executed by the image processing apparatus. [Figure 13] It is a flowchart showing the depth correction processing for chromatic pixels.

Embodiments for Carrying Out the Invention

[0012] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For clarity of explanation, redundant explanations will be omitted where necessary.

[0013] Figure 1 is a block diagram showing the configuration of the imaging system 1000 according to this embodiment. The imaging system 1000 comprises an RGB sensor (imaging unit) 200, a distance measuring sensor 300, and an image processing device 100. The imaging system 1000 is an information processing system capable of capturing a subject using the RGB sensor 200 and the distance measuring sensor 300, and performing predetermined image processing in the image processing device 100.

[0014] The RGB sensor 200 is a sensor capable of detecting color information by photographing a subject. The color information is, for example, RGB values ​​defined in the sRGB space. The color information may also be in the form of RGB values ​​defined in the Adobe® RGB space, or Lab values ​​defined in the Lab space.

[0015] The RGB sensor 200 performs processing that includes at least one of the following: automatic white balance (AWB) processing and automatic exposure (AE) processing, and then photographs the subject. AWB processing automatically determines the light source conditions of the subject and reproduces the appropriate color state. The RGB sensor 200 may perform AWB processing at all times or at the time of preparation for shooting. AE processing controls the aperture value, shutter speed, etc., based on the brightness information of the shooting field of view, and maintains a constant brightness in the captured image.

[0016] In this embodiment, the RGB sensor 200 performs both AWB and AE processing and outputs the captured image to the image processing device 100. The captured image may be a still image or a moving image. The RGB sensor 200 also outputs the RGB values ​​of each pixel in the captured image to the image processing device 100. The RGB sensor 200 is, for example, a color still camera (e.g., an RGB camera) or a color video camera.

[0017] The distance measuring sensor 300 is a sensor capable of detecting depth information by photographing a subject. Depth information is information indicating the depth of the subject in the depth direction. Depth information is represented, for example, using a depth value that indicates the distance from the distance measuring sensor 300. The depth value may be the distance from the distance measuring sensor 300 to the subject expressed in physical units such as millimeters, or it may be a normalized representation of the distance in the range of 0 to 1.

[0018] The distance measuring sensor 300 outputs a depth value corresponding to each pixel of the captured image to the image processing device 100. The distance measuring sensor 300 is, for example, a ToF sensor or a stereo camera. However, it is not limited to these, and various sensors capable of detecting the distance between the distance measuring sensor 300 and the subject may be used as the distance measuring sensor 300.

[0019] The image processing device 100 is an information processing device that acquires RGB values ​​and depth values ​​from the RGB sensor 200 and the distance measuring sensor 300 and performs image processing. The image processing device 100 comprises a detection unit 110, an acquisition unit 120, a color determination unit 130, a correction unit 140, and a storage unit 180.

[0020] The detection unit 110 acquires the captured image from the RGB sensor 200 and detects the subject area from the captured image. Figure 2 shows an example of a captured image P that includes multiple subjects. The subjects can be anything, such as people, animals, or vehicles. Captured image P includes a dog, a bicycle, and a truck as subjects. In this embodiment, an example of captured image P including three subjects is shown, but the number of subjects is not limited to three. There may be only one subject.

[0021] The subject area is the region in the captured image P that contains the subject to be detected. The subject area may, for example, represent the area within the bounding rectangle of the subject. As shown in Figure 2, the detection unit 110 detects subject areas 50a to 50c corresponding to the dog, bicycle, and truck, respectively.

[0022] For detecting the subject area, well-known object detection techniques may be used. For example, the detection unit 110 uses a deep neural network (DNN) pre-trained to detect subjects included in the captured image P to detect subjects. As object detection algorithms, for example, Faster R-CNN (Region-based Convolutional Neural Network), YOLO (You Only Look Once), or SSD (Single Shot Multibox Detector) may be used. Not limited to these, the detection unit 110 may use any method to detect the subject area. In the following explanation, subject areas 50a to 50c may be collectively referred to simply as "subject area."

[0023] The detection unit 110 assigns an identification number to each subject area detected from the captured image P. For example, the detection unit 110 assigns identification numbers such as "1", "2", and "3" to dogs, bicycles, and trucks. In this embodiment, the aforementioned "50a", "50b", and "50c" are used as identification numbers.

[0024] Returning to Figure 1, we continue the explanation. The acquisition unit 120 acquires color information and depth information for each pixel included in the subject area detected by the detection unit 110, and arranges them into an array. In this embodiment, the color information is the RGB value output from the RGB sensor 200, and the depth information is the depth value output from the distance measuring sensor 300. Figure 3 shows an example of the arrangement of RGB values ​​and depth values ​​in the subject area 50a. In the example in Figure 3, a 15-row, 7-column array is shown as an example.

[0025] Furthermore, the acquisition unit 120 acquires AWB information related to AWB processing and AE information related to AE processing from the detection unit 110. The acquisition unit 120 may also acquire AWB information and AE information based on the captured image.

[0026] Returning to Figure 1, we continue the explanation. The color determination unit 130 determines the color of each pixel based on the color information acquired by the acquisition unit 120. For example, the color determination unit 130 first determines whether each pixel is black or not. The color determination unit 130 determines whether each pixel is black or not by comparing the RGB values ​​of each pixel with a predetermined threshold, for example. The color determination unit 130 may include grays ranging from exact black (#000000) to a predetermined threshold (e.g., #0C0C0C) as "black" when determining whether each pixel is black or not. If the color determination unit 130 determines that a pixel is not black, it identifies what color that pixel is. The color determination unit 130 compares the RGB values ​​of each pixel with a predetermined threshold to determine whether the color of each pixel is red, blue, green, magenta, yellow, or cyan. Similarly, the color determination unit 130 may determine whether a pixel is white or not. Similar to the case of black, the color determination unit 130 may include shades of gray ranging from pure white (#ffffff) to a predetermined threshold (e.g., #e2e2e2) as "white" to determine whether each pixel is white or not. Any threshold can be used for determining each color. The color determination unit 130 stores the determination results in the storage unit 180, associating them with each pixel.

[0027] In the following, colors that are neither white nor black may be referred to as "chromatic colors." Examples of chromatic colors include red, blue, green, magenta, yellow, and cyan. Furthermore, pixels with chromatic colors may be referred to as "chromatic pixels." Additionally, white pixels may be referred to as "white pixels," and black pixels as "black pixels."

[0028] The correction unit 140 corrects the depth information of each pixel based on the judgment result of the color determination unit 130. The correction unit 140 performs depth value correction processing on each pixel in order, starting from the upper left pixel of the subject area 50a shown in Figure 3. Specifically, the correction unit 140 performs correction processing in the order of d[0][0], d[0][1], d[0][2], ..., d[0][6], d[1][0], d[1][1], .... Once the correction processing is completed up to d

[14] [6] in the lower right, the correction unit 140 performs correction processing on subject areas 50b and 50c in the same manner as subject area 50a.

[0029] The correction unit 140 performs different correction processing depending on the color of the pixel to be corrected. Specifically, the correction unit 140 performs different correction processing depending on whether the pixel to be corrected is a white pixel, a black pixel, or a chromatic pixel. Specifically, the correction unit 140 does not correct the depth information if the pixel to be corrected is a white pixel, and corrects the depth information in the case of a black pixel or a chromatic pixel using the method described below.

[0030] <Correction method for chromatic pixels> First, I will explain the correction method for chromatic pixels. The correction unit 140 corrects the depth information of the pixel to be corrected based on correction information associated with the color of the pixel to be corrected, if the pixel to be corrected is a chromatic pixel. Here, the correction information includes information related to the correction of depth information. In this embodiment, a correction table 181 pre-stored in the storage unit 180 is used as the correction information. However, the correction information may be provided in any format.

[0031] The correction table 181, for example, associates the color determined by the color determination unit 130 with the correction value of the depth information. The correction table 181 may be provided for each color, or it may be provided so that correction of multiple colors is possible with a single table. In addition, multiple correction tables 181 may be provided depending on the characteristics of the RGB sensor 200. Furthermore, different correction tables 181 may be provided depending on whether the shooting environment is indoors or outdoors.

[0032] Furthermore, the correction unit 140 further corrects the pixel depth information depending on whether the shooting environment of the captured image is indoors or outdoors. The correction unit 140 detects the color temperature and exposure amount of the shooting environment according to the AWB information and AE information, and determines whether the shooting environment is indoors or outdoors based on these. The correction unit 140 offsets the correction amount from the correction table 181 according to the shooting environment and further corrects the depth value in the chromatic pixels.

[0033] <Correction method for black pixels> Next, we will explain the correction method for black pixels. If the pixel to be corrected is a black pixel, the correction unit 140 corrects the depth information of the pixel to be corrected based on the depth information of a nearby chromatic pixel. Specifically, if the pixel to be corrected is a black pixel, the correction unit 140 discards the depth information of the pixel to be corrected acquired by the acquisition unit 120. The correction unit 140 corrects the depth information of the pixel to be corrected by interpolating the discarded depth information of the pixel to be corrected based on the depth information of a nearby chromatic pixel. Note that "discarding" may include not only discarding the original value, but also using an interpolated value while keeping the original value stored.

[0034] There are several correction methods for black pixels, depending on the interpolation method for discarded depth information, as described below. The correction unit 140 may select and execute one of these patterns, or it may select and execute several of them. In the following explanation, pixels corresponding to positions 1, 2, ..., n will be referred to as pixels 1, 2, ..., n, and the depth values ​​corresponding to each pixel will be expressed as D1, D2, ..., Dn.

[0035] (First correction method) First, let's explain the first correction method for black pixels. In the first correction method, the correction unit 140 interpolates the depth value of the pixel to be corrected based on the depth values ​​of the neighboring chromatic pixels in each of the four directions (up, down, left, and right) of the black pixel.

[0036] Figure 4 is an explanatory diagram of the first correction method for black pixels. The array 60a shown in Figure 4 is an example of a part of the subject area described above, and the same applies to subsequent figures. Array 60a includes pixels 0, 1, 2, ..., 8 within the subject area. In array 60a, pixel 4 is a black pixel. Pixels 0-3 and 5-8, which are in the vicinity of pixel 4, are chromatic pixels. The colors such as red, blue, and yellow shown in these chromatic pixels are examples, and the same applies to subsequent figures.

[0037] The correction unit 140 identifies the nearest chromatic pixels in the four directions of up, down, left, and right of pixel 4. In the example in Figure 4, the correction unit 140 identifies chromatic pixels 1, 3, 5, and 7 adjacent to pixel 4 in the directions of up, left, right, and down of pixel 4. The correction unit 140 discards the depth value of pixel 4 acquired by the acquisition unit 120 and interpolates the discarded depth value based on the depth values ​​of the identified chromatic pixels 1, 3, 5, and 7. Specifically, the correction unit 140 interpolates the depth value of pixel 4 such that the average of the depth values ​​D1, D3, D5, and D7 of the four identified pixels 1, 3, 5, and 7 becomes the depth value D4 of pixel 4.

[0038] The interpolated D4 is expressed as shown in equation (1) below. D4 = (D1 + D3 + D5 + D7) / 4 ... (1) Furthermore, while it is desirable for the correction unit 140 to interpolate the depth value of a black pixel based on the depth values ​​of chromatic pixels adjacent to the black pixel as described above, it is not limited to this, and may also interpolate the depth value of a black pixel based on the depth values ​​of chromatic pixels in a nearby range.

[0039] (Variation of the first correction method) In the first correction method described above, the correction unit 140 may perform interpolation represented by equation (1) using the depth values ​​of chromatic pixels that are not adjacent to black pixels.

[0040] Figure 5 illustrates a modified version of the first correction method for black pixels. The array 60b shown in Figure 5 contains pixels 10, 11, 12, ..., 30. In array 60b, pixels 18 to 21 are black pixels. Pixels near these black pixels that are not black pixels are chromatic pixels.

[0041] For example, suppose the correction unit 140 performs correction processing on the pixel 18. Pixels 11, 17, 19, and 25 are adjacent to pixel 18 in the directions above, to the left, to the right, and below, respectively. Here, we will identify pixels 11, 17, 19, and 25 in the same way as in the first correction method described above, and interpolate the depth value of pixel 18 using their depth values. Since pixel 19, which is adjacent to pixel 18 to the right, is a black pixel, using the depth value of pixel 19 may reduce the accuracy of the interpolated depth value.

[0042] Therefore, if the adjacent pixel used for interpolation is a black pixel, the correction unit 140 identifies the nearest chromatic pixel in the direction in which the adjacent pixel is located, and interpolates the depth value using the depth value of the identified chromatic pixel. Accordingly, instead of pixel 19, the correction unit 140 identifies the nearest chromatic pixel 22 to the right of pixel 18, and interpolates the depth value of pixel 18 using the depth value of pixel 22.

[0043] The correction unit 140 discards the depth value of pixel 18 acquired by the acquisition unit 120 and interpolates the depth value of pixel 18 using the depth values ​​D11, D17, D22, and D25 of the identified pixels 11, 17, 22, and 25.

[0044] The depth value D18 of pixel 18 after interpolation is expressed by the following equation (2). D18=(D11+D17+D22+D25) / 4 ···(2)

[0045] Similarly, let's assume that the correction unit 140 performs correction processing on pixel 19. Pixel 19 is a black pixel, and the adjacent pixel 18 to its left is also a black pixel. Since the correction unit 140 performs correction processing according to the array order, when correcting pixel 19, the depth value of pixel 18 is the interpolated depth value D18 expressed by equation (2) above. Therefore, when the correction unit 140 performs correction on pixel 19, it can interpolate the depth value of pixel 19 using the depth value D18 of the adjacent pixel 18 to its left.

[0046] Note that the pixel 20 adjacent to the right of pixel 19 is a black pixel, and its depth value is the one before correction. Therefore, in the rightward direction, the correction unit 140 identifies the nearest chromatic pixel 22, similar to the case of pixel 18 described above, and performs interpolation using the depth value of the identified pixel 22. Thus, the correction unit 140 identifies pixels 12, 18, 22, and 26. The correction unit 140 discards the depth value of pixel 19 acquired by the acquisition unit 120 and interpolates the depth value of pixel 19 using the depth values ​​D12, D18, D22, and D26 of the identified pixels 12, 18, 22, and 26.

[0047] The depth value D19 of pixel 19 after interpolation is expressed by the following equation (3). D19=(D12+D18+D22+D26) / 4 ···(3)

[0048] (Second correction method) Next, we will explain the second correction method for black pixels. In the second correction method, the correction unit 140 interpolates the depth value of the pixel to be corrected based on the depth value of the pixel with the lowest brightness among the chromatic pixels in the vicinity of the black pixel.

[0049] Figure 6 is an explanatory diagram of the second correction method for black pixels. The array 60c shown in Figure 6 contains pixels 40, 41, ..., 48 within the subject area. In array 60c, pixel 44 is a black pixel. Pixels 40-43 and 45-48, which are in the vicinity of pixel 44, are chromatic pixels of colors 1-4 and 5-8, respectively. Alternatively, pixels 40-43 may be black pixels after depth values ​​have been interpolated. In the following explanation, the eight pixels surrounding pixel 44, 40-43 and 45-48, will be collectively referred to as "8 pixels."

[0050] First, the correction unit 140 calculates the brightness value Y for each of the eight pixels based on the RGB values ​​of each pixel. The brightness value Y can be calculated using the following formula (4) based on each of the RGB components. Note that the coefficients multiplied by each component are not limited to those shown below and may be changed as appropriate. Y=0.2126×R+0.7152×G+0.0722×B (4)

[0051] Based on the calculation result of equation (4), the correction unit 140 identifies the pixel with the smallest brightness value Y among the 8 pixels and interpolates the depth value of the pixel to be corrected using the depth value of the identified pixel. In the example in Figure 6, it is assumed that pixel 48 has the smallest brightness value Y among the 8 pixels. Therefore, the correction unit 140 identifies pixel 48. The correction unit 140 discards the depth value of pixel 44 acquired by the acquisition unit 120 and interpolates the depth value of pixel 44 by using the depth value D48 of the identified pixel 48 as the depth value D44 of pixel 44.

[0052] In this way, the depth value of the black pixel to be corrected can be interpolated using the depth value of the pixel with the lowest brightness, i.e., the pixel closest to black.

[0053] Although this explanation uses eight pixels in the vicinity of the pixel to be corrected, it is not limited to this. The correction unit 140 may interpolate the depth value using the brightness values ​​of more or fewer pixels than eight. For example, if the brightness values ​​of eight nearby pixels cannot be obtained, such as when a black pixel is located at the edge of the subject area, the correction unit 140 may perform interpolation using the number of brightness values ​​that can be obtained.

[0054] (Third correction method) Next, we will explain the third correction method for black pixels. In the third correction method, the correction unit 140 interpolates the depth values ​​of multiple pixels to be corrected within the black pixel region based on the respective depth values ​​of multiple chromatic pixels adjacent to the black pixel region in which multiple black pixels are consecutive.

[0055] Figures 7 and 8 are explanatory diagrams of a third correction method for black pixels. The array 60d shown in Figures 7 and 8 includes pixels 50, 51, ..., 58 within the subject area. In array 60d, pixels 52-56 are black pixels, forming a black pixel region b1. Pixels 50 and 51 are located to the left of the black pixel region b1, and pixels 57 and 58 are located to the right. Pixels 50, 51, 57, and 58 are chromatic pixels.

[0056] Furthermore, for the sake of explanation, in the following, we will refer to the pixels adjacent to the left of the black pixel region b1 as adjacent pixels c1, and the pixels adjacent to the right as adjacent pixels c2. In the examples in Figures 7 and 8, adjacent pixel c1 is pixel 51, and adjacent pixel c2 is pixel 57.

[0057] First, the correction unit 140 calculates the brightness level of each pixel in the array 60d. The brightness level indicates the magnitude of brightness at each pixel. Here, the brightness value Y of each pixel calculated by equation (4) above is used as the brightness level. However, the brightness level may be calculated using other methods.

[0058] The luminance level curve 70 shown in Figure 7 is an example of the luminance level of each pixel in the array 60d. The correction unit 140 generates the luminance level curve 70 of the array 60d from the luminance level of each pixel. Furthermore, the portion of the luminance level curve 70 corresponding to the black pixel region b1 is designated as the luminance level curve 70b1. Note that the Y shown in Figure 7 max and Y min These are the maximum and minimum values ​​of the luminance value Y in the black pixel region b1, respectively.

[0059] The depth value curve 80 shown in Figure 8 is an example of the depth value of each pixel in the array 60d. The correction unit 140 generates the depth value curve 80 of the array 60d from the depth value of each pixel. The correction unit 140 identifies adjacent pixels c1 and c2 adjacent to the black pixel region b1 in the left or right direction. In the example of FIG. 7, the pixel 51 adjacent to the left side of the pixel 52 at the left end of the black pixel region b1 is identified as the adjacent pixel c1, and the pixel 57 adjacent to the black pixel 56 at the right end of the black pixel region b1 is identified as the adjacent pixel c2.

[0060] The correction unit 140 obtains the difference range of the depth values of the adjacent pixels c1 and c2. If the depth values of the adjacent pixels c1 and c2 are represented by D c1 and D c2 respectively, the difference range is represented by (D c1 - D c2 ). The correction unit 140 discards the depth values of each pixel in the black pixel region b1 acquired by the acquisition unit 120, and interpolates the depth values of each pixel in the black pixel region b1 using the luminance level curve 70b1 shown in FIG. 7 so that the depth values of each pixel fall within the difference range.

[0061] Let the position of each pixel in the black pixel region b1 be n, the luminance value of the black pixel to be corrected in the black pixel region b1 be Yn, the minimum luminance value of the black pixel region b1 be Y min、 the maximum luminance value of the black pixel region b1 be Y max the depth value of the adjacent pixel c1 be D c1 and the depth value of the adjacent pixel c2 be D c2 . Then, the depth value Dn of the black pixel to be corrected in the black pixel region b1 is represented by the following formula (5). Dn = (Yn - Y min ) × { (D c1 - D c2 ) / (Y max - Y min )} + D c2 ···(5)

[0062] The correction unit 140 interpolates the depth values of the black pixels 52 to 56 in the black pixel region b1 using the above formula (5). By doing so, the correction unit 140 adjusts the depth values of each pixel in the black pixel region b1 according to the depth values D c1 and D c2The interpolation can be performed so that the values ​​fall within a certain range. In Figure 8, the depth values ​​of each pixel within the interpolated black pixel region b1 are shown by the depth value curve 80b1.

[0063] In Figures 7 and 8, a 1-row, 9-column array was used as the array 60d for explanation, but the method is not limited to this. The third correction method may also be applied to arrays having multiple rows. Therefore, the third correction method may be used even when the black pixel region b1 is formed across multiple rows. In this case, the correction unit 140 identifies, for example, the chromatic pixels adjacent to the leftmost or rightmost black pixel in the black pixel region b1 as adjacent pixels c1 and c2, respectively.

[0064] Furthermore, if the black pixel region b1 is formed across multiple rows, the correction unit 140 may identify adjacent pixels c1 and c2 from pixels in the vertical direction of the black pixel region b1, rather than from pixels in the horizontal direction. For example, adjacent pixels c1 and c2 may be identified so as to sandwich the black pixel region b1 from above or below. The correction unit 140 is not limited to these methods and may identify adjacent pixels c1 and c2 by other means.

[0065] (A variation of the third correction method) Figure 9 illustrates a modified example of the third correction method. For example, suppose that lighting is provided on the side of the distance measuring sensor 300, and the lighting illuminates the subject. In this case, the closer the distance measuring sensor 300 is to the subject, the stronger the illumination light hits it, and the further away the distance is, the weaker the illumination light hits it.

[0066] Areas of the subject that are strongly illuminated are located closer to the distance sensor 300. Therefore, pixels with high brightness values ​​have smaller depth values ​​compared to pixels with low brightness values. On the other hand, areas of the subject that are weakly illuminated are located further away from the distance sensor 300. Therefore, pixels with low brightness values ​​have larger depth values ​​compared to pixels with high brightness values. Using this, the correction unit 140 may further correct the depth values ​​of the correction target pixels that have been corrected by the third correction method, according to the brightness level of the correction target pixels.

[0067] The correction unit 140 further corrects the depth value of each black pixel in the black pixel region b1 based on, for example, the brightness level curve 70b1 shown in Figure 7 and the depth value curve 80b1 shown in Figure 8. The correction unit 140 further corrects the depth value of each black pixel by converting the depth value of each pixel so that the depth value of each pixel decreases as the brightness value of each pixel increases. The depth value curve 81b1 shown in Figure 9 is an example of the converted depth value curve. In this way, the depth value of the black pixels can be corrected based on the relative brightness levels of each pixel in the black pixel region b1.

[0068] (Fourth correction method) Next, we will explain the fourth correction method for black pixels. In the fourth correction method, the correction unit 140 interpolates the depth value of the pixel to be corrected using spline interpolation to interpolate between data points.

[0069] Figures 10 and 11 are explanatory diagrams of the fourth correction method for black pixels. Figure 10 is a graph of the depth values ​​of array 60e, which has a black pixel region b2 within the subject area. In Figure 10, the horizontal axis shows the coordinates corresponding to each pixel of array 60e, and the vertical axis shows the depth value of each pixel. In the figure, white circles represent data for chromatic pixels, and black circles represent data for black pixels.

[0070] The correction unit 140 identifies the black pixel region b2 and discards the depth values ​​of each pixel within the black pixel region b2 acquired by the acquisition unit 120. The correction unit 140 interpolates the discarded depth values ​​using spline interpolation based on the depth values ​​of neighboring chromatic pixels. The correction unit 140 interpolates the depth values ​​of each pixel so that the depth values ​​of each pixel in the black pixel region b2 are continuous with those of neighboring chromatic pixels.

[0071] Figure 11 shows the data after spline interpolation. The interpolated black pixel data is shown by shading. In this way, the depth value of a black pixel can be interpolated using the depth values ​​of neighboring chromatic pixels. However, the correction unit 140 may also interpolate the depth value of black pixels using well-known interpolation methods such as linear interpolation or polynomial interpolation.

[0072] The above are the first to fourth correction methods for black pixels. The correction unit 140 may select one of these correction methods and perform the correction, or it may combine several of these correction methods to perform the correction. For example, the correction unit 140 may select one of the first to fourth correction methods according to the number of black pixels included in the black pixel area and the shape of the black pixel area, and perform the correction on the target pixel. However, the correction unit 140 may select a correction method according to any conditions, such as the number of chromatic pixels in the vicinity of a black pixel or the ratio of black pixels to chromatic pixels in the entire subject area.

[0073] The correction unit 140 performs corrections on all arrays within the subject area 50a for chromatic pixels and black pixels using the correction method described above. When the correction in the subject area 50a is completed, the correction unit 140 then performs the correction process on the subject area 50b. When the processing is completed for the subject areas 50a to 50c detected in the captured image P, the correction process is terminated.

[0074] Returning to Figure 1, we continue the explanation. The memory unit 180 is a storage device for storing various types of information. The memory unit 180 pre-stores the correction table 181 described above. The memory unit 180 also stores programs for realizing each function of the image processing device 100.

[0075] Next, the processes performed by the image processing device 100 will be explained using Figure 12. Figure 12 is a flowchart showing the processes performed by the image processing device 100. Each functional unit used below corresponds to that in Figure 1. Figures 2 to 11 will also be referred to as appropriate in the explanation.

[0076] The detection unit 110 acquires the captured image from the RGB sensor 200 (S11). Here, we will explain assuming that the captured image P shown in Figure 2 has been acquired. It is assumed that the captured image P has been subjected to AWB processing and AE processing by the RGB sensor 200.

[0077] The detection unit 110 detects subjects from the captured image P using well-known object detection techniques (S12). In the example shown in Figure 2, the detection unit 110 detects subject regions 50a to 50c corresponding to the subjects, namely the dog, bicycle, and truck. The subject regions 50a to 50c are, for example, the areas within the bounding rectangle of each subject.

[0078] Next, the detection unit 110 assigns an identification number to each detected subject (S13). The detection unit 110 assigns the identification numbers "50a", "50b", and "50c" to the subject areas, which include dogs, bicycles, and trucks, respectively.

[0079] The acquisition unit 120 acquires color information and depth information of pixels included in the subject area 50a and arranges them as shown in Figure 3 (S14). The color information is the RGB value output from the RGB sensor 200, and the depth information is the depth value output from the distance measuring sensor 300. The image processing device 100 performs the following processing on the correction target pixels in the subject area 50a according to this arrangement order.

[0080] The color determination unit 130 determines the color of each pixel based on the color information acquired by the acquisition unit 120 (S15). For example, the color determination unit 130 compares the RGB values ​​of each pixel with a predetermined threshold to determine whether the pixel is black or not. If the color determination unit 130 determines that the pixel is not black, it identifies what color the pixel is. The color determination unit 130 compares the RGB values ​​of each pixel with a predetermined threshold to determine whether the color of each pixel is red, blue, green, magenta, yellow, or cyan. Similarly, the color determination unit 130 may determine whether the pixel is white or not. Any threshold may be used for determining each color. The color determination unit 130 stores the determination results in the storage unit 180 in association with each pixel.

[0081] The correction unit 140 sequentially performs correction processing on each pixel, starting from the top-left pixel of the subject area 50a, treating each pixel as a target pixel for correction. The correction unit 140 obtains the judgment result from the color judgment unit 130 and performs different correction processing according to the color of each pixel (S16). If the target pixel for correction is a white pixel (white in S16), the correction unit 140 proceeds to step S19 without performing any correction.

[0082] If the pixel to be corrected is a black pixel ("black" in S16), the correction unit 140 performs depth correction processing on the black pixel using the first to fourth correction methods for black pixels, as explained with reference to Figures 4 to 11 (S17). Specifically, the correction unit 140 discards the depth value of the black pixel acquired by the acquisition unit 120 and corrects the depth value of the black pixel by interpolating the discarded depth value based on the depth value of a chromatic pixel in the vicinity of the black pixel. Since each correction method has already been explained, a detailed explanation will be omitted here and the explanation will be simplified as appropriate.

[0083] When using the first correction method, the correction unit 140 interpolates the depth value of a pixel based on the depth value of the nearest chromatic pixel in each of the four directions (up, down, left, and right) of the black pixel. For example, as explained with reference to Figure 4, the correction unit 140 interpolates the depth value of the pixel to be corrected using the depth values ​​of chromatic pixels adjacent to it in the up, down, left, and right directions. Alternatively, as explained with reference to Figure 5, the correction unit 140 may interpolate using the depth values ​​of chromatic pixels that are not adjacent to the black pixel.

[0084] When using the second correction method, the correction unit 140 interpolates the depth value of a pixel based on the depth value of the pixel with the lowest brightness among the chromatic pixels in the vicinity of the black pixel. As explained with reference to Figure 6, the correction unit 140 calculates the brightness value of each of the eight pixels surrounding the pixel to be corrected. The correction unit 140 interpolates so that the depth value of the pixel with the lowest brightness becomes the depth value of the pixel to be corrected.

[0085] When using the third correction method, the correction unit 140 interpolates the depth values ​​of multiple pixels based on the respective depth values ​​of multiple chromatic pixels adjacent to a black pixel region containing multiple black pixels. As explained with reference to Figure 7, the correction unit 140 calculates the luminance value of each pixel included in the array and generates a luminance level curve. The correction unit 140 identifies the chromatic pixels on both sides of the black pixel region and finds the difference in depth values ​​of the identified chromatic pixels. As explained with reference to Figure 8, the correction unit 140 interpolates the depth values ​​of each pixel in the black pixel region so that the depth values ​​of the black pixels fall within the difference range. Furthermore, as explained with reference to Figure 9, the correction unit 140 may further correct the depth values ​​by converting the depth values ​​of the pixels to be corrected so that the greater the luminance value of the black pixels in the black pixel region, the smaller the depth value becomes.

[0086] When using the fourth correction method, the correction unit 140 interpolates the depth values ​​of the pixels to be corrected using spline interpolation to interpolate between data. As explained with reference to Figures 10 and 11, the correction unit 140 discards the uncorrected depth values ​​in the black pixel region and interpolates the depth values ​​in the black pixel region so that the depth values ​​of each pixel are continuous with those of neighboring chromatic pixels.

[0087] Returning to Figure 12, we continue the explanation. If the pixel to be corrected is a chromatic pixel ("chromatic" in S16), the correction unit 140 performs depth correction processing on the chromatic pixel (S18). Here, we will explain the depth correction process for chromatic pixels using Figure 13. Figure 13 is a flowchart of the depth correction process for chromatic pixels.

[0088] The correction unit 140 refers to a correction table 181 associated with the color of the pixels to be corrected and corrects the depth value of the pixels to be corrected based on the correction table 181 (S21). Next, the correction unit 140 determines whether the shooting environment of the captured image P is indoors or outdoors (S22). For example, the correction unit 140 acquires AWB and AE information performed by the RGB sensor 200 and makes this determination based on the color temperature and exposure of the captured image P.

[0089] The correction unit 140 offsets the depth value correction amount according to the above determination result (S23). As a result, the correction unit 140 can further correct the depth value according to whether the shooting environment of the captured image P is indoors or outdoors, in addition to the correction using the pre-provided correction table 181.

[0090] Returning to Figure 4, we continue the explanation. The correction unit 140 determines whether or not correction processing has been performed on all sequences in the subject area 50a (S19). If there are any sequences that have not been processed (NO in S19), the correction unit 140 returns to step S16 and repeats the subsequent processing. If correction processing has been performed on all sequences in the subject area 50a (YES in S19), the process proceeds to the next step.

[0091] Next, the correction unit 140 determines whether or not image processing has been performed on all subjects detected in the captured image P (S20). If image processing has been performed on all subjects (YES in S20), the process ends. If there are unprocessed subjects (NO in S20), the process returns to step S14 and the subsequent processes are repeated.

[0092] As described above, in the shooting system 1000 according to this embodiment, the RGB sensor 200 and the distance measuring sensor 300 capture the subject and output color information and depth information to the image processing device 100. In the image processing device 100, the detection unit 110 detects the subject area from the captured image, and the acquisition unit 120 acquires the color information and depth information of the pixels included in the subject area and arranges them.

[0093] The color determination unit 130 determines the color of each pixel based on color information, and the correction unit 140 corrects the depth information of each pixel based on the determination result. The correction unit 140 can perform different correction processing depending on the color of the pixel. For example, if the pixel to be corrected is a chromatic pixel, the correction unit 140 corrects the depth information of the pixel to be corrected based on the correction table 181 associated with the color of the pixel. The correction unit 140 also determines whether the shooting environment of the captured image is indoors or outdoors, and further corrects the depth information of the pixel to be corrected according to the determination result.

[0094] Furthermore, if the pixel to be corrected is a black pixel, the correction unit 140 can select one or more correction methods from a plurality of correction methods to correct the depth information of the black pixel. For example, in the first correction method, the correction unit 140 corrects the depth information of the pixel to be corrected based on the depth information of chromatic pixels in the vicinity of the pixel to be corrected. The correction unit 140 identifies chromatic pixels adjacent to the black pixel above, below, left, and right, and performs the correction using their depth information. Alternatively, if the adjacent pixels are black pixels, the correction unit 140 may identify the nearest non-adjacent chromatic pixel and use its depth information. In this way, the depth information of a black pixel can be corrected using the depth information of chromatic pixels surrounding the pixel to be corrected.

[0095] In the second correction method, the correction unit 140 identifies the pixel with the lowest brightness among the chromatic pixels near the black pixel and corrects the depth information of the pixel to be corrected based on its depth information. In this way, the depth information of black pixels can be corrected using the depth information of chromatic pixels that are closer to black.

[0096] Furthermore, in the third correction method, the correction unit 140 can correct the depth information of multiple pixels to be corrected based on the depth information of multiple chromatic pixels adjacent to a black pixel region containing multiple black pixels. For example, the correction unit 140 calculates the difference in depth values ​​of chromatic pixels adjacent to pixels at both ends of the black pixel region and corrects the depth value of the pixels to be corrected so that it falls within that difference range. In this way, the depth information of the black pixel region can be contained within the difference range of the depth information of the adjacent chromatic pixels. Furthermore, the correction unit 140 can perform further correction according to the brightness level of the pixels within the black pixel region. The correction unit 140 can correct the depth value of each pixel by estimating whether the subject is in front of or behind the pixel based on the brightness level of each pixel.

[0097] In the fourth correction method, the correction unit 140 can correct the depth information of the pixels to be corrected using a well-known interpolation method such as spline interpolation. In this way, if the depth value of the black pixel region and the depth value of the surrounding chromatic pixels are discontinuous, the depth value of the black pixel region can be corrected so that they become continuous.

[0098] Thus, with the shooting system 1000 according to this embodiment, different correction processing can be performed depending on the color of the pixel to be corrected, making it possible to appropriately correct depth information according to the color of the subject.

[0099] Note that the configuration of the imaging system 1000 shown in Figure 1 is merely an example. Each component of the imaging system 1000 may be configured using a device that integrates multiple components. For example, some or all of the functions of the image processing device 100, the RGB sensor 200, and the distance measuring sensor 300 may be integrated into the same device. For example, one or both of the RGB sensor 200 and the distance measuring sensor 300 may be built into the image processing device 100. Furthermore, each functional unit in the image processing device 100 may be processed in a distributed manner using multiple devices.

[0100] The image processing device 100 may also include an output unit (not shown) for outputting the captured image P before or after the correction process. The output unit may be, for example, a display. The output unit may also have an input function such as a touch panel. Furthermore, the image processing device 100 may be configured to output a 3D image based on depth values.

[0101] <Example Hardware Configuration> Each functional component of the image processing device 100, the RGB sensor 200, and the distance measuring sensor 300 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and a program to control them). For example, this disclosure can also be implemented by having a CPU (Central Processing Unit) execute a computer program.

[0102] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include computer-readable mediums or physical storage mediums such as RAM (random-access memory), ROM (read-only memory), flash memory, SSD (solid-state drive), or other memory technologies, CD-ROMs, DVDs (digital versatile discs), Blu-ray® discs, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable mediums or communication mediums such as electrical, optical, acoustic, or other forms of propagating signals.

[0103] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, in the above description, the correction unit 140 was described as not performing any correction processing on the depth value of white pixels, but this is not limited to this. The correction unit 140 may perform some kind of correction on the depth value of white pixels. [Explanation of symbols]

[0104] 100 Image Processing Devices 110 Detection unit 120 Acquisition Department 130 Color judgment section 140 Correction section 180 Storage section 181 Correction Table 200 RGB sensor (imaging unit) 300 Distance measuring sensor 1000 shooting system P Photographed image 50a~50c Subject area 60a~60e Sequence 70, 70b1 Brightness Level Curve 80, 80b1, 81b1 depth value curves 0-8 pixels 10-30 pixels 40-48 pixels 50-58 pixels c1, c2 adjacent pixels b1, b2 Black pixel regions

Claims

1. A detection unit that detects the subject area from the captured image, An acquisition unit that acquires color information and depth information for each pixel included in the subject area, A color determination unit that determines the color of the pixel based on the aforementioned color information, The system includes a correction unit that corrects the depth information of the pixels based on the determination result of the color determination unit, If the subject area includes a black pixel region in which multiple black pixels determined to be black by the color determination unit are consecutive, the correction unit identifies multiple chromatic pixels adjacent to the black pixel region as adjacent pixels, and corrects the depth information of each black pixel in the black pixel region using a brightness level curve generated from the brightness levels of each black pixel so that it falls within the difference range of the depth information of the multiple adjacent pixels. Image processing device.

2. The correction unit identifies a plurality of chromatic pixels adjacent to the black pixel region as adjacent pixels, such that they sandwich the black pixel region from different directions. The image processing apparatus according to claim 1.

3. The correction unit further corrects the depth information of the corrected black pixel according to the brightness level of the black pixel. The image processing apparatus according to claim 1 or 2.

4. The correction unit corrects the depth information of the corrected black pixel so that it decreases as the brightness level of the black pixel increases. The image processing apparatus according to claim 3.

5. A detection step to detect the subject area from the captured image, An acquisition step of acquiring color information and depth information for each pixel included in the subject area, A color determination step in which the color of the pixel is determined based on the color information, The correction step includes correcting the depth information of the pixel based on the determination result in the color determination step, In the correction step, if the subject area includes a black pixel region in which multiple black pixels determined to be black in the color determination step are consecutive, multiple chromatic pixels adjacent to the black pixel region are identified as adjacent pixels, and the depth information of each black pixel in the black pixel region is corrected using a brightness level curve generated from the brightness levels of each black pixel so that it falls within the difference range of the depth information of the multiple adjacent pixels. Image processing methods.

Images