Image processing device and image processing method

Abstract

An image processing device (10) according to the present disclosure comprises a first image acquisition unit for acquiring a first image captured by a first camera for detecting light having a first wavelength, a second image acquisition unit for acquiring a second image captured by a second camera for detecting light having a second wavelength and measuring a distance, a difference extraction unit (15) for extracting a difference between a visible light image and an infrared light image, a shadow region detection unit (16) for detecting a shadow region which is a region in which a shadow does not occur in the visible light image but does occur in the infrared light image, and a correction unit (19) for correcting distance information of a pixel in the shadow region measured by the second camera, on the basis of distance information of a pixel in the vicinity of the shadow region.

Description

Image processing device and image processing method

[0001] The present disclosure relates to an image processing device and an image processing method.

[0002] A technology for correcting a captured image by performing predetermined image processing on the image captured by a camera is known. For example, Patent Document 1 discloses an image processing device including a first image acquisition unit for acquiring a first image, a second image acquisition unit for acquiring a second image, and a correction unit for correcting second information of a first region of the first image using second information of a region different from the first region. The first and second images are images generated from light of first and second wavelengths, respectively. Patent Document 1 discloses visible light images and infrared light images as examples of the first and second images. The image processing device reproduces dark regions (shadow regions) of a subject generated by, for example, flash illumination by correcting color information of the dark regions.

[0003] The ToF (Time of Flight) method is also known as one of the techniques for measuring the distance (depth) from an image capture device to a subject. A ToF sensor using the ToF method, for example, irradiates an infrared distance measurement light toward the subject and receives the distance measurement light reflected from the subject with an infrared image sensor. The ToF sensor can calculate the distance between the subject and the image capture device by detecting the time difference between the irradiation and reception of the light for each pixel.

[0004] In recent years, imaging devices equipped with such a ToF sensor and an RGB sensor capable of capturing RGB images have become known. By capturing an image of a subject using such an imaging device, an infrared image can be obtained in addition to a visible light image.

[0005] JP 2016-039409 A

[0006] When capturing images using the above-described imaging device, the visible light image does not capture reflected light of infrared wavelengths, resulting in different appearances of the subject between the visible light image and the infrared light image. Furthermore, because the positions of the ToF sensor and the RGB sensor are different, parallax occurs between the visible light image and the infrared light image. Therefore, when comparing the visible light image and the infrared light image, shadows may appear in different regions. For example, there may be a case where no shadow appears in region a in the visible light image, but a shadow appears in region a in the infrared light image. In such a case, a 3D image obtained by combining the visible light image and the infrared light image may have a problem in which the region a appears as an unnatural depression. The technology disclosed in Patent Document 1 does not take such a problem into consideration.

[0007] In view of the above-described problems, an object of the present disclosure is to provide an image processing device and an image processing method that are capable of appropriately correcting distance information of a shadow area that appears in an image.

[0008] The image processing device according to the present disclosure includes: a first image acquisition unit that acquires a first image captured by a first camera that detects light of a first wavelength; a second image acquisition unit that acquires a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure distance; a difference extraction unit that extracts the difference between the first image and the second image; a shadow area detection unit that detects a shadow area, which is an area in the first image where no shadow occurs but an area in the second image where a shadow occurs, based on the difference; and a correction unit that corrects distance information of pixels in the shadow area measured by the distance measurement sensor based on distance information of pixels near the shadow area.

[0009] The image processing method according to the present disclosure is performed by a computer, which executes the following steps: a first image acquisition step of acquiring a first image captured by a first camera that detects light of a first wavelength; a second image acquisition step of acquiring a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure distance; a difference extraction step of extracting the difference between the first image and the second image; a shadow area detection step of detecting a shadow area, which is an area in the first image where no shadow occurs but an area in the second image where a shadow occurs, based on the difference; and a correction step of correcting distance information of pixels in the shadow area measured by the distance measurement sensor based on distance information of pixels near the shadow area.

[0010] The image processing device according to the present disclosure includes: a first image acquisition unit that acquires a first image captured by a first camera that detects light of a first wavelength; a second image acquisition unit that acquires a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure distance; a difference extraction unit that extracts the difference between the first image and the second image; a shadow area detection unit that detects a shadow area in the second image based on the difference; and a correction unit that corrects distance information of pixels in the shadow area measured by the distance measurement sensor using a correction value based on distance information of multiple pixels near the shadow area.

[0011] The image processing method according to the present disclosure is performed by a computer, which executes the following steps: a first image acquisition step of acquiring a first image captured by a first camera that detects light of a first wavelength; a second image acquisition step of acquiring a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure distance; a difference extraction step of extracting a difference between the first image and the second image; a shadow area detection step of detecting a shadow area in the second image based on the difference; and a correction step of correcting distance information of pixels in the shadow area measured by the distance measurement sensor using a correction value based on distance information of multiple pixels near the shadow area.

[0012] The image processing device and image processing method according to the present disclosure make it possible to appropriately correct distance information of a shadow area that appears in an image.

[0013] 14 is a block diagram showing the configuration of an imaging system according to a first embodiment. FIG. 15 is a diagram showing a pre-correction image. FIG. 16 is a diagram showing the pre-correction image as viewed from the lateral direction (positive X-axis direction). FIG. 17 is a flowchart showing shadow region extraction processing for extracting a shadow region. FIG. 18 is a diagram explaining the angle of view adjustment processing in the angle of view adjustment unit. FIG. 19 is a diagram showing a visible light image after grayscale conversion processing. FIG. 19 is a diagram explaining the processing of a difference extraction unit and a shadow region detection unit. FIG. 19 is a flowchart showing the correction processing performed by an image processing device. FIG. 20 is a diagram showing an example of depth values ​​before and after correction processing. FIG. 21 is a diagram showing 3D images before and after correction processing. FIG. 22 is a diagram showing the pre-correction image and the corrected image as viewed from the lateral direction (positive X-axis direction). FIG. 23 is a block diagram showing the configuration of an imaging system according to a second embodiment. FIG. 24 is a flowchart showing the correction processing performed by the image processing device. FIG. 25 is a diagram showing an example of depth values ​​before and after correction processing. FIG. 26 is an enlarged view of region r1 in FIG. 25. FIG. 27 is a diagram showing 3D images before and after correction processing.

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals. For clarity of explanation, duplicated explanations will be omitted as necessary.

[0015] First Embodiment (Configuration of Image Capturing System 100) First, a first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an image capturing system 100 according to this embodiment. The image capturing system 100 includes an RGB sensor 20 as a first camera, a distance measuring sensor 30 as a second camera, and an image processing device 10. The first and second cameras are image capturing devices capable of detecting light in different wavelength bands and capturing images.

[0016] The first camera is an imaging device that detects light of a first wavelength and captures an image. The first camera outputs a first image generated by the imaging to the image processing device 10. In this embodiment, the description will be given using visible light as an example of light of the first wavelength.

[0017] The second camera is an imaging device that detects light of the second wavelength and captures an image. The second camera outputs a second image generated by the imaging to the image processing device 10. In this embodiment, infrared light is used as an example of light of the second wavelength. The second camera also has a distance measurement sensor that measures the distance to the subject. The distance measurement sensor measures the distance to the subject for each pixel and outputs distance information, which is the measurement result, to the image processing device 10.

[0018] The photographing system 100 is an information processing system capable of performing image processing on photographed images generated by the first and second cameras in the image processing device 10. The photographing system 100 can be realized as a camera equipped with each functional unit.

[0019] (Configuration of RGB sensor 20) The RGB sensor 20 is an example of the first camera described above. The RGB sensor 20 is an imaging device that detects visible light and captures an image. The RGB sensor 20 captures an image of a subject using visible light and detects color information for each pixel. The color information is, for example, RGB information defined in sRGB space. The color information may be in the form of RGB values ​​defined in Adobe (registered trademark) RGB space or Lab values ​​defined in Lab space. In this embodiment, the description will be given using RGB values ​​as RGB information.

[0020] In this embodiment, an RGB camera will be used as an example of the RGB sensor 20. The RGB sensor 20 outputs a visible light image generated by capturing an image to the image processing device 10. The visible light image may be a still image or a moving image. The RGB sensor 20 also outputs the RGB values ​​of each pixel of the visible light image to the image processing device 10.

[0021] (Configuration of Distance Measuring Sensor 30) The distance measuring sensor 30 is an example of the second camera described above. The distance measuring sensor 30 is an imaging device that detects infrared light to capture an image. The distance measuring sensor 30 captures an image of a subject using infrared light and detects distance information indicating the distance between the distance measuring sensor 30 and the subject for each pixel. The distance information is, for example, depth information indicating the depth of the subject in the depth direction. The depth information is expressed, for example, using a depth value indicating the distance from the distance measuring sensor 30. The depth value may be a physical unit such as millimeters that represents the distance from the distance measuring sensor 30 to the subject, or may be a normalized representation of the distance in the range of 0 to 1. In this embodiment, the description will be given using a depth value as the distance information.

[0022] In addition, in this embodiment, a ToF sensor will be used as an example of the distance measurement sensor 30. The distance measurement sensor 30 irradiates an object with LD (Laser Diode) light in the infrared region and receives the LD light reflected by the object with an infrared image sensor. The distance measurement sensor 30 calculates the distance between the object and the distance measurement sensor 30 by detecting the time difference between the irradiation and reception of the light for each pixel. The distance measurement sensor 30 is not limited to a ToF sensor, and various sensors capable of detecting the distance between the object and the distance measurement sensor 30 may be used.

[0023] The distance measurement sensor 30 outputs a depth value for each pixel to the image processing device 10. The distance measurement sensor 30 also generates an infrared light image using infrared light and outputs it to the image processing device 10. The infrared light image may be a still image or a moving image. The distance measurement sensor 30 also outputs brightness information for each pixel of the infrared light image to the image processing device 10.

[0024] (Configuration of image processing device 10) Next, the configuration of the image processing device 10 will be described. Detailed processing in each functional unit will be described later using flowcharts. The image processing device 10 is an information processing device that acquires information about visible light images and infrared light images from the RGB sensor 20 and the distance measurement sensor 30, and performs predetermined image processing.

[0025] The image processing device 10 includes a processor, a memory, and a storage device (not shown). The storage device stores a computer program that implements the processing according to this embodiment. The processor can then load the computer program from the storage device into the memory and execute the computer program. This allows the processor to realize various functions, which will be described later.

[0026] As shown in FIG. 1, the image processing device 10 includes a visible light image acquisition unit (first image acquisition unit) 11, an infrared light image acquisition unit (second image acquisition unit) 12, an angle of view adjustment unit 13, a conversion unit 14, a difference extraction unit 15, a shadow area detection unit 16, and a correction unit 19.

[0027] The visible light image acquisition unit 11 acquires a visible light image from the RGB sensor 20. The visible light image acquisition unit 11 functions as a first image acquisition unit that acquires a first image captured by a first camera that detects light of a first wavelength. The visible light image acquisition unit 11 also acquires the RGB values ​​of each pixel of the visible light image from the RGB sensor 20.

[0028] The infrared image acquisition unit 12 acquires an infrared image from the distance measurement sensor 30. The infrared image acquisition unit 12 functions as a second image acquisition unit that acquires a second image captured by a second camera of the distance measurement sensor that detects light of a second wavelength to measure distance. The infrared image acquisition unit 12 also acquires depth values ​​and brightness information for each pixel of the infrared image from the distance measurement sensor 30.

[0029] The angle-of-view adjustment unit 13 performs an angle-of-view adjustment process to align the angles of view of the visible light image and the infrared light image. The RGB sensor 20 and the distance measurement sensor 30 may have different numbers of pixels and angles of view. Therefore, the angle-of-view adjustment unit 13 adjusts the angles of view of the visible light image and the infrared light image by performing trimming and clipping processes on the visible light image and the infrared light image so that the angles of view of the visible light image and the infrared light image are aligned.

[0030] For example, the angle-of-view adjusting unit 13 adjusts the angle of view of the visible light image or the infrared light image so that the angle of view of the other image matches that of the visible light image or the infrared light image, whichever has the smaller number of pixels or angle of view. In this way, the angle-of-view adjusting unit 13 matches the angles of view of the visible light image and the infrared light image.

[0031] The conversion unit 14 performs a conversion process in which the RGB values ​​of each pixel of the visible light image are converted into grayscale to obtain luminance information. In this way, the conversion unit 14 obtains luminance information of each pixel of the visible light image. Well-known techniques may be used for the conversion process to grayscale.

[0032] In this embodiment, the conversion unit 14 converts the RGB values ​​of the visible light image into grayscale, but this is not limited to this. The conversion unit 14 may select and convert either the first or second image depending on the wavelength of light used in the first and second cameras. Alternatively, the conversion unit 14 may convert both the first and second images. Furthermore, the conversion unit 14 may perform color conversion processing other than grayscale conversion.

[0033] The difference extraction unit 15 extracts the difference between the visible light image and the infrared light image. Specifically, the difference extraction unit 15 extracts the difference in luminance information between the visible light image and the infrared light image. The difference extraction unit 15 may extract the difference using the luminance information after conversion by the conversion unit 14. In this embodiment, the difference extraction unit 15 extracts the difference between the luminance information of the visible light image after grayscale conversion by the conversion unit 14 and the luminance information of the infrared light image acquired by the infrared light image acquisition unit 12. The difference extraction unit 15 generates a difference extraction image indicating the difference at each pixel.

[0034] Based on the differences extracted by the difference extraction unit 15, the shadow region detection unit 16 detects shadow regions, which are regions where no shadow occurs in the visible light image and where a shadow occurs in the infrared light image because the infrared light emitted from the distance measurement sensor 30 is blocked by an object and does not reach the infrared light imaging element. First, the shadow region detection unit 16 performs difference enhancement processing to enhance the difference in the difference extraction image generated by the difference extraction unit 15. For example, the shadow region detection unit 16 binarizes the difference using a predetermined threshold value to generate a difference enhanced image. The threshold value may be set in advance or may be changed as appropriate depending on the magnitude of the extracted difference, etc.

[0035] The shadow region detection unit 16 identifies a shadow region based on the difference-enhanced image. For example, the shadow region detection unit 16 determines that a pixel with a difference of "1" is a pixel in a shadow region. The shadow region detection unit 16 also determines that a pixel with a difference of "0" is not a pixel in a shadow region. Furthermore, as long as it is possible to identify an area where a shadow occurs in the infrared light image, a shadow region does not necessarily have to be an area where no shadow occurs in the visible light image. The shadow region detection unit 16 may detect a shadow region in the infrared light image using a brightness level threshold, taking advantage of the fact that the brightness levels of shadows occurring in the visible light image and those in the infrared light image differ.

[0036] In this way, the shadow region detection unit 16 detects, as a shadow region, a region formed by pixels determined to be pixels of a shadow region. Note that, when a predetermined number or more of pixels determined to be pixels of a shadow region are consecutive, the shadow region detection unit 16 may detect the plurality of pixels as a shadow region.

[0037] The correction unit 19 corrects the depth values ​​of pixels in the shadow region measured by the distance measurement sensor 30 based on the depth values ​​of pixels near the shadow region. Hereinafter, pixels near the shadow region may be referred to as "neighboring pixels." Neighboring pixels may be located in any of four directions (left, right, top, bottom) relative to the shadow region. The number of neighboring pixels may be set appropriately depending on factors such as the distance from the boundary of the shadow region.

[0038] When a pixel to be corrected is located in a shadow region, the correction unit 19 corrects the depth value of the pixel to be corrected based on the depth values ​​of neighboring pixels. Specifically, when the pixel to be corrected is located in a shadow region, the correction unit 19 discards the depth value of the pixel to be corrected acquired by the infrared image acquisition unit 12. The correction unit 19 corrects the depth value of the pixel to be corrected by interpolating the discarded depth value of the pixel to be corrected based on the depth values ​​of neighboring pixels. Note that "discarding" may include not only discarding the original value, but also using an interpolated value while keeping the original value stored.

[0039] For example, assume that pixels adjacent to a shadow region are used as neighboring pixels. The correction unit 19 corrects the depth value of the correction target pixel using the depth values ​​of the pixels adjacent to the shadow region. In the following description, pixels adjacent to a shadow region may be referred to as "adjacent pixels." Adjacent pixels may be located in any of four directions (up, down, left, or right) of a pixel located on the boundary of the shadow region.

[0040] (Processing of the Image Capturing System 100) Next, the processing performed by the image capturing system 100 will be specifically described. For the sake of explanation, an example in which the correction processing according to this embodiment is not performed will be described with reference to Fig. 2. Fig. 2 is a diagram showing a pre-correction image c1 as an example of a 3D image obtained by combining a visible light image and an infrared light image.

[0041] The right-handed XYZ coordinate system shown in Figure 2 is used for the convenience of explaining the positional relationships of components in a 3D image. The XYZ coordinate system is common to the following drawings. The imaging system 100 images a subject from the positive side toward the negative side of the Z axis.

[0042] As shown in FIG. 2, multiple blocks are placed on the XY plane. Furthermore, as indicated by the arrow in the figure, a character figurine is placed on one of the blocks in the center of the figure. As indicated by the dashed line area in the figure, a shadow area, which is a depression that does not actually exist, has occurred behind the character. As shown in FIG. 5 (described later), no shadow area has occurred in the visible light image. However, as described above, because visible light images do not capture reflections of light in the infrared wavelength range, shadows may occur in different areas between the visible light image and the infrared light image.

[0043] In this example, an accurate depth value was not acquired for the part of the character's shadow when photographed by the distance measurement sensor 30. Therefore, in the pre-correction image c1, which is simply a combination of the visible light image and the infrared light image, this part appears as an unnatural depression behind the character.

[0044] 3 is a diagram of the pre-correction image c1 as viewed from the side (positive X-axis direction). As shown in the figure, a depression appears on the negative Z-axis side in the pre-correction image c1. In this embodiment, the depth values ​​of pixels corresponding to such shadow areas can be corrected by performing the image processing described below.

[0045] Next, the processing executed by the image processing device 10 according to this embodiment will be specifically described with reference to Figures 4 to 11. The functional units used below correspond to those in Figure 1.

[0046] (Shadow Region Extraction Processing) First, the shadow region extraction processing performed by the image processing device 10 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the shadow region extraction processing for extracting a shadow region.

[0047] First, the visible light image acquisition unit 11 and the infrared light image acquisition unit 12 acquire a visible light image and an infrared light image, respectively (S101). The visible light image acquisition unit 11 acquires a visible light image from the RGB sensor 20, and the infrared light image acquisition unit 12 acquires an infrared light image from the distance measurement sensor 30. The infrared light image acquisition unit 12 also acquires a depth value of each pixel in the infrared light image from the distance measurement sensor 30. Hereinafter, the acquired images will be referred to as a visible light image a1 and an infrared light image b1, respectively.

[0048] The visible light image acquisition unit 11 acquires the RGB values ​​of each pixel included in the visible light image a1 and arranges them into an array. The infrared light image acquisition unit 12 acquires the depth values ​​of each pixel included in the infrared light image b1 and arranges them into an array. As a result, the visible light image acquisition unit 11 and the infrared light image acquisition unit 12 each acquire a pixel array of m rows and n columns (m and n are natural numbers).

[0049] In this embodiment, the image processing device 10 performs various processes described below in a scanning direction from the upper left pixel to the lower right pixel of the array. The image processing device 10 performs processing from left to right within the same row of the array. Note that the processing order is not limited to this. For example, the image processing device 10 may perform processing in a scanning direction from the upper right pixel to the lower left pixel. Furthermore, the image processing device 10 may scan and perform processing in a column direction, not limited to the row direction.

[0050] The angle-of-view adjusting unit 13 performs an angle-of-view adjustment process to align the angles of view of the visible light image a1 and the infrared light image b1 (S102). For example, the angle-of-view adjusting unit 13 adjusts the angles of view of both the visible light image a1 and the infrared light image b1 so as to match the smaller one of the visible light image a1 and the infrared light image b1 in terms of the number of pixels or the angle of view.

[0051] 5 is a diagram illustrating the angle-of-view adjustment process performed by the angle-of-view adjustment unit 13. For example, assume that the pixel size of the visible light image a1 shown in the diagram is 1280 × 960 pixels, and the pixel size of the infrared light image b1 is 640 × 480 pixels. In the diagram, the angle of view b10 of the infrared light image b1 is indicated by a dashed dotted line.

[0052] In this case, the angle-of-view adjustment unit 13 adjusts the angle of view of the visible light image a1 so that the pixel size of the visible light image a1 matches that of the infrared light image b1, which has a smaller pixel size. For example, the angle-of-view adjustment unit 13 trims the visible light image so that the angle of view of the visible light image matches the angle of view b10 of the infrared light image b1. In this way, the angle-of-view adjustment unit 13 can match the angles of view of the visible light image a1 and the infrared light image b1, thereby making it possible to match pixels between the visible light image a1 and the infrared light image b1.

[0053] 4, the conversion unit 14 then performs grayscale conversion processing to convert the visible light image a1 into grayscale (S103). The conversion unit 14 converts the RGB values ​​of the visible light image a1 into grayscale, and sets the luminance information obtained after conversion as the luminance information of the visible light image a1.

[0054] The conversion unit 14 may perform the grayscale conversion process using a well-known technique. For example, the conversion unit 14 may perform the process using OpenCV, an open-source image processing library. However, the conversion unit 14 is not limited to this, and any other method may be used.

[0055] 6 is a diagram showing a visible light image a2 obtained after grayscale conversion of the visible light image a1. The conversion unit 14 acquires luminance information for each pixel of the visible light image a2. The luminance information indicates the luminance of each pixel. In this embodiment, the luminance information will be described using luminance values ​​expressed in 256 gradations. For example, a pixel with a luminance value of 0 represents black, and a pixel with a luminance value of 255 represents white.

[0056] In the conversion unit 14, the luminance values ​​in the visible light image can be obtained by grayscaling the visible light image, and in the next step S104, the luminance values ​​of the visible light image can be compared with the luminance values ​​of the infrared light image.

[0057] 4, the difference extraction unit 15 performs a difference extraction process to extract the difference between the visible light image a2 and the infrared light image b1 after grayscale conversion (S104). The difference extraction unit 15 compares the brightness values ​​of each pixel in each image to extract the difference.

[0058] 7 is a diagram illustrating the processing of the difference extraction unit 15 and the shadow region detection unit 16. The upper part of Fig. 7 shows a visible light image a2 and an infrared light image b1. The difference extraction unit 15 extracts the difference between the visible light image a2 and the infrared light image b1 and generates a difference extraction image c2 shown in the middle part of the figure.

[0059] Next, the shadow region detection unit 16 detects a shadow region, which is a region where no shadow occurs in the visible light image a2 and a region where a shadow occurs in the infrared light image b1, based on the extracted difference (S105). Specifically, the shadow region detection unit 16 performs a difference enhancement process to enhance the difference using the difference extraction image c2. For example, the shadow region detection unit 16 binarizes the difference using a predetermined threshold value to generate the shadow region image c3 shown in the lower part of Figure 7. In the figure, the region where the difference is enhanced is shown as a shadow region in white.

[0060] The shadow region detection unit 16 identifies a shadow region using the shadow region image c3. For example, the shadow region detection unit 16 determines that a pixel with a difference of "1" is a pixel in a shadow region. The shadow region detection unit 16 also determines that a pixel with a difference of "0" is not a pixel in a shadow region. The shadow region detection unit 16 detects, as a shadow region, an area formed by pixels determined to be pixels in a shadow region.

[0061] The shadow region detection unit 16 may detect a plurality of pixels as a shadow region when a predetermined number or more of pixels are determined to be shadow region pixels. The predetermined number can be set appropriately depending on the image. The shadow region detection unit 16 identifies the position of the shadow region using coordinates on the pixel array. For example, the shadow region detection unit 16 identifies the coordinates of the pixel corresponding to the start position of the shadow region and the coordinates of the pixel corresponding to the end position of the shadow region.

[0062] (Correction Process) Next, the correction process after the shadow region is identified will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the correction process performed by the image processing device 10 to correct the depth values ​​of pixels in the shadow region.

[0063] The correction unit 19 scans the pixels in the shadow region image c3 in order, starting from the top left pixel, and performs the following correction process on the correction target pixel. Note that the correction process includes determining whether or not to correct the correction target pixel. Also, as described above, the scanning direction is not limited to this.

[0064] First, the correction unit 19 determines whether or not the correction process has been performed on all pixels of the shadow region image c3 (S201). If the correction process has been performed on all pixels (YES in S201), the process ends.

[0065] If there is an unprocessed pixel in the shadow region image c3 (NO in S201), the correction unit 19 determines whether the pixel to be corrected is a pixel in the shadow region (S202). If the pixel to be corrected is a pixel in the shadow region (YES in S202), the correction unit 19 corrects the depth value of the pixel to be corrected using the depth value of an adjacent pixel (S203). Here, the correction unit 19 scans from the pixel on the left to the pixel on the right, so the adjacent pixel indicates the pixel immediately before entering the shadow region.

[0066] If the pixel to be corrected is not a pixel in a shadow region (NO in S202), the process returns to step S201 and the corrector 19 repeats the process from step S201 onwards until the correction process is completed for all pixels in the shadow region image c3.

[0067] 9 is a diagram showing an example of depth values ​​before and after correction processing. Here, the depth values ​​of the shadow region s1 and pixels near the shadow region s1 are shown as an example. The horizontal axis indicates the coordinates corresponding to each pixel in the pixel array, and the vertical axis indicates the depth value of each pixel. The upper row of FIG. 9 shows the depth values ​​before correction, and the lower row shows the depth values ​​after correction.

[0068] 9, shadow region s1 includes pixels p2, p3, p4, .... Pixel p2 is the pixel at the start position of shadow region s1. Correction unit 19 corrects the depth values ​​of pixels p2, p3, p4, ... of shadow region s1 measured by infrared image acquisition unit 12 based on the depth values ​​of neighboring pixels near shadow region s1.

[0069] The neighboring pixel is, for example, adjacent pixel p1 adjacent to shadow region s1. The correction unit 19 corrects the depth value of the pixel in shadow region s1 using the depth value of adjacent pixel p1. For example, assume that the pixel to be corrected is pixel p2. The correction unit 19 corrects the depth value of pixel p2 using the depth value of adjacent pixel p1. The correction unit 19 discards the depth value of pixel p2 before correction and corrects the depth value of pixel p2 by interpolating the discarded depth value based on the depth value of adjacent pixel p1.

[0070] If the depth value of the adjacent pixel p1 is D1 and the depth value of the corrected pixel p2 is D2, D2 is expressed by the following equation (1): D2=D1 (1)

[0071] The correction unit 19 also corrects the correction target pixels p3, p4, ... in the same manner as described above when the correction target pixels are pixels p3, p4, .... Accordingly, the correction unit 19 corrects the depth values ​​of pixels p3, p4, ... included in shadow region s1 to D1. As a result, as shown in the lower part of Figure 9, the depth values ​​of each pixel in shadow region s1 after correction become constant. Furthermore, after correction processing, the depth values ​​of the pixels in shadow region s1 become smaller than before correction.

[0072] 10 and 11 are diagrams showing 3D images before and after correction processing. In FIGS. 10 and 11, the upper row shows pre-correction image c1, and the lower row shows post-correction image c4. As shown in FIG. 10, the post-correction image c4 has a smaller depression in the shadow area behind the character compared to the pre-correction image c1. In this way, the post-correction image c4 has been corrected to a state closer to a more natural flat surface compared to the pre-correction image c1.

[0073] Figure 11 shows the pre-correction image c1 and the corrected image c4 shown in Figure 10 as viewed from the side (positive X-axis direction). In the pre-correction image c1, a depression occurred in the flat surface portion, causing it to sink toward the negative Z-axis direction. However, in the corrected image c4, this depression has been reduced. Figure 11 also shows that the correction process has reduced the depth values ​​of the pixels in the shadow region compared to before correction.

[0074] As described above, the imaging system 100 according to this embodiment includes the RGB sensor 20 (first camera), the distance measurement sensor 30 (second camera), and the image processing device 10. The RGB sensor 20 detects visible light (light of a first wavelength) and generates a visible light image (first image). The distance measurement sensor 30 detects infrared light (light of a second wavelength), measures the distance to the subject, and generates an infrared light image (second image).

[0075] In the image processing device 10, a visible light image acquisition unit 11 acquires a visible light image, and an infrared light image acquisition unit 12 acquires an infrared light image. A field-of-view adjustment unit 13 adjusts the field of view of the visible light image and the infrared light image. A conversion unit 14 converts the RGB values ​​of the visible light image into grayscale to acquire a luminance value.

[0076] The difference extraction unit 15 extracts the difference between the visible light image and the infrared light image using their respective brightness values. The shadow region detection unit 16 detects a shadow region, which is a region where no shadow occurs in the visible light image but a shadow occurs in the infrared light image, based on the difference. The correction unit 19 corrects the depth values ​​(distance information) of pixels in the shadow region measured by the distance measurement sensor 30 based on the depth values ​​of pixels near the shadow region. For example, the correction unit 19 corrects the depth values ​​of pixels in the shadow region using the depth values ​​of pixels adjacent to the shadow region.

[0077] As described above, the imaging system 100 according to this embodiment can detect pixels in a shadow region and appropriately correct the depth values ​​of the pixels in the shadow region using the depth values ​​of pixels near the shadow region. This allows the imaging system 100 to superimpose the corrected depth values ​​on the visible light image, thereby achieving the correction effect described above in the generated 3D image. This allows the imaging system 100 to reduce unnatural depressions in the 3D image and generate a more natural 3D image.

[0078] Furthermore, currently, cameras equipped with ToF sensors are required to be designed to minimize the distance between multiple irradiators that irradiate distance measurement light and a light-receiving unit that receives distance measurement light reflected from a subject in order to reduce the occurrence of shadow areas. For example, the occurrence of shadow areas can be reduced by arranging multiple irradiators at equal intervals around a circle centered on the light-receiving unit near the light-receiving unit. In contrast, the imaging system 100 according to this embodiment can appropriately correct shadow areas even with a single irradiator that irradiates distance measurement light, thereby reducing the occurrence of shadow areas regardless of the distance between the irradiator and the light-receiving unit. This allows for flexible design without being restricted by the placement of the irradiators and the light-receiving unit.

[0079] 1 is merely an example. Each component of the imaging system 100 may be configured using a device that integrates multiple components. For example, some or all of the functions of the image processing device 10, the RGB sensor 20, and the distance measurement sensor 30 may be integrated into the same device. For example, one or both of the RGB sensor 20 and the distance measurement sensor 30 may be configured to be built into the image processing device 10. Furthermore, each functional unit in the image processing device 10 may be distributed and processed using multiple devices.

[0080] The image processing device 10 may also include an output unit (not shown) for outputting the captured image or 3D image before or after correction processing. The output unit is, for example, a display. The output unit may have an input function such as a touch panel.

[0081] <Embodiment 2> Next, embodiment 2 will be described. Embodiment 2 is a modification of embodiment 1. In embodiment 1, the image processing device 10 corrects the depth value of a pixel in a shadow region using the depth value of a single pixel near the shadow region. As described with reference to FIGS. 10 and 11 , performing the correction process in embodiment 1 can reduce unnatural depressions in shadow regions. However, when correction is performed using the depth value of a single pixel in this way, the correction traces may be noticeable in the corrected 3D image, resulting in an unnatural image.

[0082] The imaging system 100a according to this embodiment uses the depth values ​​of multiple pixels near a shadow region to determine a correction value for correcting the depth value of the pixel in the shadow region, and then corrects the depth value of the shadow region using the determined correction value, thereby achieving a higher correction effect.

[0083] The following description will focus on differences from the first embodiment, and will simplify the description of overlapping parts as appropriate. Similar to the first embodiment, the present embodiment will be described assuming that the light of the first wavelength is visible light and the light of the second wavelength is infrared light.

[0084] (Configuration of Image Capturing System 100a) The image capturing system 100a according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the image capturing system 100a. The image capturing system 100a includes an RGB sensor 20, a distance measuring sensor 30, and an image processing device 10a. The RGB sensor 20 and the distance measuring sensor 30 are the same as those in the first embodiment described with reference to Fig. 1, and therefore will not be described here.

[0085] (Configuration of Image Processing Device 10a) In addition to the configuration of the image processing device 10 of the first embodiment, the image processing device 10a includes a sampling unit 17 and a correction value determination unit 18. The sampling unit 17 performs a depth value sampling process. For example, the sampling unit 17 samples the depth values ​​of multiple pixels near a shadow region.

[0086] The correction value determination unit 18 determines a correction value for correcting the depth value of a pixel in a shadow region based on the depth values ​​of multiple neighboring pixels near the shadow region. The correction value indicates a correction amount for offsetting the depth value before correction. The multiple neighboring pixels may be located in any of four directions (left, right, top, bottom) relative to the shadow region. Furthermore, the positions of the multiple neighboring pixels do not have to be consecutive.

[0087] For example, the correction value determination unit 18 determines the correction value based on the depth values ​​of multiple pixels surrounding the correction target pixel. Note that, in the following description, the pixels surrounding the correction target pixel may be referred to as "surrounding pixels." The correction value determination unit 18 determines the correction value taking into account the position of the correction target pixel in the shadow region.

[0088] For example, a shadow region may have a first correction region close to the boundary with the shadow region and a second correction region farther from the boundary than the first correction region. In the first correction region, at least one of the multiple surrounding pixels is near the shadow region, and in the second correction region, multiple surrounding pixels are within the shadow region. For example, the first correction region is close to the start position of the shadow region, and the second correction region is farther from the start position of the shadow region than the first correction region. The shadow region may also have a third correction region close to the end position of the shadow region. The first, second, and third correction regions can be represented as the entrance region, middle region, and exit region of the shadow region, respectively. The first, second, and third correction regions may be set appropriately depending on the scanning direction.

[0089] The correction value determination unit 18 determines the correction value using different methods depending on whether the pixel to be corrected is in the first correction area or the second correction area, as will be described in detail later.

[0090] The correction unit 19 corrects the depth values ​​of the pixels in the shadow region using the correction value determined by the correction value determination unit 18. The correction unit 19 acquires the corrected depth value by adding the correction value to the pre-correction depth value of the pixel to be corrected.

[0091] (Correction Processing) Next, the processing performed by the image capturing system 100a according to this embodiment will be specifically described with reference to Fig. 13. Fig. 13 is a flowchart showing the correction processing performed by the image processing device 10a.

[0092] The process by which the image processing device 10a acquires the visible light image and the infrared light image from the RGB sensor 20 and the distance measurement sensor 30 and extracts the shadow region is the same as the process in steps S101 to S105 described using Fig. 4, and therefore will not be described here. Also, as in the first embodiment, the image processing device 10a scans the shadow region image c3 in order from the upper left pixel to the lower right.

[0093] After generating the shadow region image c3, the image processing device 10a determines whether correction processing has been performed on all pixels in the shadow region image c3 (S301). If processing has been performed on all pixels in the shadow region image c3 (YES in S301), the processing ends. If processing has not been performed on all pixels in the shadow region image c3 (NO in S301), the image processing device 10a determines whether the pixel to be corrected is a pixel in a shadow region (S302).

[0094] First, a case where the pixel to be corrected is not a pixel in a shadow region (NO in S302) will be described. The sampling unit 17 performs a depth value sampling process and calculates the average value of the sampled depth values ​​(S305). The sampling unit 17 samples the depth values ​​of multiple pixels near the shadow region, for example. Here, the sampling process will be described with reference to FIGS. 14 and 15.

[0095] FIG. 14 is a diagram showing an example of depth values ​​before and after correction processing according to this embodiment. Here, depth values ​​of pixels in shadow region s1 and the vicinity of shadow region s1 are shown as an example. The horizontal axis indicates the coordinates corresponding to each pixel in the pixel array, and the vertical axis indicates the depth value of each pixel. The upper part of the diagram shows the depth values ​​before correction, and the lower part shows the depth values ​​after correction. The image processing device 10a scans from the left side (coordinate 0 side) to the right side (coordinate 70 side) of FIG. 14 and sequentially performs correction processing on pixels to be corrected.

[0096] 14, the shadow region s1 has, from the left, an entrance region (first correction region) s11 close to the boundary with the vicinity of the shadow region s1, an intermediate region (second correction region) s12 away from the boundary, and an exit region (third correction region) s13 close to the boundary with the vicinity of the shadow region s1. In the entrance region s11 and the exit region s13, at least one of the multiple peripheral pixels is located near the shadow region s1. In addition, in the intermediate region s12, multiple peripheral pixels are located within the shadow region s1.

[0097] Figure 15 is an enlarged view of region r1 in Figure 14. Region r1 includes the entrance region s11 and part of the intermediate region s12 in shadow region s1. Here, assume that the pixel to be corrected is pixel p13. Pixel p13 is not a pixel included in shadow region s1. Pixel p13 is also a neighboring pixel of shadow region s1, adjacent to pixel p14 located on the boundary of shadow region s1.

[0098] The sampling unit 17 samples the depth values ​​of pixels p11 to p13, including multiple pixels p11 and p12 surrounding pixel p13, and calculates the average value AVG of the depth values ​​of pixels p11 to p13. In FIG. 15, AVG is indicated by a dashed line.

[0099] In this way, when the pixel to be corrected is not in a shadow region, the sampling unit 17 obtains the average value of the depth values ​​of multiple pixels including the pixel to be corrected. This allows the sampling unit 17 to obtain the average value of the depth values ​​of multiple neighboring pixels immediately before entering the shadow region s1. As will be described below, this average value is used in determining a correction value in the correction value determination unit 18. Note that, although an example has been used in which the sampling unit 17 samples the depth values ​​of three pixels p11 to p13, the number of samples is not limited to this.

[0100] Returning to FIG. 13 , the case where the correction target pixel is a pixel in a shadow region (YES in S302) will be described. The correction value determination unit 18 determines a correction value depending on the position of the correction target pixel within the shadow region s1 (S303). Specifically, the correction value determination unit 18 identifies which of the entrance region s11 to exit region s13 the correction target pixel is included in, and determines a correction value depending on the identified correction region. The correction value determination unit 18 determines the correction value using the average value AVG calculated by the sampling unit 17 based on the depth values ​​of multiple surrounding pixels surrounding the correction target pixel.

[0101] The correction value determination unit 18 may specify in advance, before the correction process, which of the entrance region s11 to the exit region s13 the correction target pixel is included in, or may specify this during the correction process.

[0102] When specifying the shadow regions in advance, for example, the correction value determiner 18 specifies the coordinates of the pixels at the start and end positions of the shadow region detected in the shadow region image c3. The correction value determiner 18 specifies one or more pixels from the start position of the shadow region as pixels of the entrance region s11, and one or more pixels from the end position as pixels of the exit region s13. The correction value determiner 18 also specifies pixels in the region between the entrance region s11 and the exit region s13 as pixels of the intermediate region s12.

[0103] Alternatively, the correction value determination unit 18 may specify the correction area according to the magnitude of change in depth value between the pixel to be corrected and the pixel immediately preceding it during the correction process. The correction value determination unit 18 compares the depth value of the pixel to be corrected with the depth value of the pixel immediately preceding the pixel to be corrected and determines whether the change in depth value is large.

[0104] For example, the correction value determination unit 18 may determine that the change in depth value is large when the difference in depth value between the pixel to be corrected and the pixel immediately preceding it is equal to or greater than a predetermined threshold. Alternatively, the correction value determination unit 18 may calculate the slope of the change in depth value and determine that the change in depth value is large when the slope is equal to or greater than a predetermined threshold. The threshold may be set in advance or may be changed depending on the image.

[0105] 15, assume that the pixel to be corrected is pixel p14. Pixel p14 is a pixel included in shadow region s1. Furthermore, pixel p14 is the pixel at the start position of shadow region s1, so the correction value determination unit 18 identifies pixel p14 as a pixel in entrance region s11.

[0106] Also, assume that the pixel to be corrected is pixel p15. Like pixel p14, pixel p15 is a pixel included in shadow region s1. The correction value determination unit 18 compares the difference between the depth value of pixel p15 and the depth value of the immediately preceding pixel p14 with a predetermined threshold, and identifies the correction region for pixel p15 based on the comparison result. Here, it is assumed that the correction value determination unit 18 has determined that there is a large change in depth value. The correction value determination unit 18 identifies pixel p15 as a pixel in entrance region s11.

[0107] Also, assume that the pixel to be corrected is pixel p16. Like pixels p14 and p15, pixel p16 is a pixel included in shadow region s1. The correction value determination unit 18 compares the difference between the depth value of pixel p16 and the depth value of the immediately preceding pixel p15 with a predetermined threshold, and identifies the correction region for pixel p16 based on the comparison result. Here, it is assumed that the correction value determination unit 18 has determined that the change in depth value is not large. The correction value determination unit 18 identifies pixel p16 as a pixel in intermediate region s12.

[0108] Similarly, for pixels p17 and onward, the correction value determination unit 18 compares the depth value of the pixel to be corrected with the depth value of the pixel immediately preceding the pixel to be corrected. Then, the correction value determination unit 18 identifies to which correction region, among the entrance region s11 to the exit region s13, the pixel to be corrected belongs.

[0109] The correction value determination unit 18 may specify the correction area using a method other than the above-described method. For example, the correction value determination unit 18 may specify the correction area to which the correction target pixel belongs by counting the number of pixels since the start of the shadow area s1. Alternatively, the correction value determination unit 18 may specify the correction area in advance based on the number of pixels in the shadow area s1, the depth value of each image, etc.

[0110] The correction value determination unit 18 determines a correction value according to the correction region. For example, the correction value determination unit 18 determines a correction value for each pixel in the entrance region s11, and determines a correction value for the intermediate region s12 using the correction value determined in the entrance region s11.

[0111] A specific explanation will be given using the above-mentioned pixels p14 to p16. First, assume that the pixel to be corrected is pixel p14. Pixel p14 is the pixel at the start position of shadow area s1. The correction value determination unit 18 determines a correction value v1 for correcting the depth value of pixel p14. If the depth value of pixel p14 is D14a, v1 is expressed by the following equation (2): v1=D14a-AVG (2)

[0112] Also, assume that the pixel to be corrected is pixel p15. Pixel p15 is not at the start position of shadow region s1 and is a pixel included in entrance region s11. The correction value determination unit 18 determines a correction value v2 for correcting the depth value of pixel p15. If the depth value of pixel p15 is D15a, v2 is expressed by the following equation (3): v2=D15a-AVG (3)

[0113] In this way, the correction value determination unit 18 can acquire an appropriate correction value by determining v2, which is different from v1, as the correction value to be used for correcting the depth value of pixel p15. This allows the correction value determination unit 18 to update the correction value to be used for correcting a correction target pixel whose depth value has a large change from the previous pixel.

[0114] Also, assume that the pixel to be corrected is pixel p16. Pixel p16 is a pixel included in intermediate region s12. Pixel p16 is also the pixel at the start position of intermediate region s12. The correction value determination unit 18 determines a correction value v3 for correcting the depth value of pixel p16. Since the depth value does not change significantly in intermediate region s12, the correction value determination unit 18 determines v2 as v3. v3 is expressed by the following equation (4): v3 = v2 (4)

[0115] The correction value determination unit 18 similarly determines correction values ​​for pixels p17 and subsequent pixels. Therefore, the correction values ​​used to correct the pixels in the intermediate region s12 are constant. However, if the depth value in the intermediate region s12 changes significantly, the correction value determination unit 18 may determine a new correction value based on the depth values ​​of multiple surrounding pixels around the pixel to be corrected.

[0116] 15, the correction value determination unit 18 may determine the correction value for the exit region s13 in the same manner as for the entrance region s11. For example, the correction value determination unit 18 may determine the correction value using depth values ​​of multiple neighboring pixels located to the right of the exit region s13. The correction value determination unit 18 may also determine the correction value for the exit region s13 using v1 and v2 determined for the entrance region s11.

[0117] In this way, the correction value determination unit 18 can determine a correction value depending on whether the pixel to be corrected is located in the entrance region s11, the intermediate region s12, or the exit region s13 of the shadow region s1. By determining different correction values ​​depending on the correction region in the correction value determination unit 18, the corrector 19 can correct the depth values ​​of the pixels in the shadow region s1 so that the boundary of the shadow region s1 is expressed smoothly.

[0118] 13 , the correction unit 19 corrects the depth values ​​of the pixels in the shadow region s1 using the correction value determined by the correction value determination unit 18 (S304). The correction unit 19 adds the correction value to the depth value before correction to obtain the corrected depth value.

[0119] For example, assume that the pixel to be corrected is pixel p14. The correction value is v1, and the depth value of pixel p14 before correction is D14a. If the depth value of pixel p14 after correction is D14b, D14b is expressed by the following equation (5): D14b=D14a+v1 (5)

[0120] Furthermore, suppose the pixel to be corrected is pixel p15. The correction value is v2, and the depth value of pixel p15 before correction is D15a. If the depth value of pixel p15 after correction is D15b, D15b is expressed by the following equation (6): D15b=D15a+v2 (6)

[0121] Also, assume that the pixel to be corrected is pixel p16. The correction value is v3, but since v2, the same correction value as for pixel p15, is used, v2 will be used below. If the depth value of pixel p16 before correction is D16a and the depth value of pixel p16 after correction is D16b, D16b is expressed by the following equation (7): D16b = D16a + v2 (7)

[0122] The correction unit 19 also performs correction on the pixels subsequent to pixel p17 included in the intermediate region s12 using v2, in the same manner as pixel p16.

[0123] As a result of the above-described correction process, the depth values ​​of the pixels in the shadow region s1 are corrected after the correction process, as shown in the lower part of Figure 14. After the correction process, the depth values ​​of the pixels in the shadow region s1 are smaller than before the correction. In the first embodiment, the correction was performed using a single depth value, so the depth value after correction was constant. However, in this embodiment, the correction can be performed while maintaining the unevenness of the depth values ​​(variations in the depth values) in the shadow region s1.

[0124] 16 is a diagram showing 3D images before and after correction processing according to this embodiment. The upper row shows a pre-correction image c5, and the lower row shows a corrected image c6. Note that the pre-correction image c5 is the same as the corrected image c4 in embodiment 1. As shown in FIG. 16, the corrected image c6 has been corrected so that the correction marks in the shadow areas behind the character are less noticeable compared to the pre-correction image c5.

[0125] In the above description, the correction value determination unit 18 determines the correction value using consecutive pixels p11 to p13 as peripheral pixels of pixel p14, which is the pixel to be corrected, but the correction value may be determined using other methods. For example, the correction value determination unit 18 may determine the correction value using two neighboring pixels in the shadow region s1 that are located at a distance from each other.

[0126] For example, suppose that pixels p21 and p22 are located near shadow region s1, sandwiching shadow region s1. The correction value determination unit 18 determines a correction value using the depth value of pixel p21 and the depth value of pixel p22. For example, the correction value determination unit 18 determines the correction value of each pixel in shadow region s1 so that the depth value of the pixel in shadow region s1 is smaller than the depth value indicated by the line connecting the depth value of pixel p21 and the depth value of pixel p22. Note that pixels p21 and p22 may be located on either side of shadow region s1, or on either side of shadow region s1. Alternatively, pixels p21 and p22 may be located on the upper and left sides of the shadow region, respectively.

[0127] As described above, the imaging system 100a according to this embodiment includes the same components as those of the imaging system 100 described in the first embodiment, as well as the sampling unit 17 and the correction value determination unit 18 in the image processing device 10a.

[0128] The sampling unit 17 performs a sampling process to sample the depth values ​​of multiple pixels near the shadow region. The correction value determination unit 18 determines a correction value for correcting the depth values ​​of the pixels in the shadow region based on the depth values ​​of multiple neighboring pixels near the shadow region. The correction unit 19 then corrects the depth values ​​of the pixels in the shadow region using the correction value determined by the correction value determination unit 18.

[0129] With this configuration, the imaging system 100a according to this embodiment can achieve the same effects as those of embodiment 1. Furthermore, rather than uniformly correcting the depth values ​​of pixels in the shadow region, the imaging system 100a effectively utilizes the depth values ​​of the shadow region before correction, and can use values ​​obtained by offsetting the depth values ​​before correction as the corrected depth values.

[0130] Furthermore, because the depth values ​​change stepwise between the entrance and exit regions of the shadow area, adjusting the correction values ​​for the pixels in these regions makes it possible to obtain a more natural image as a result of the correction. This eliminates unnatural depressions in the 3D image obtained by combining the visible light image and the infrared light image, allowing for the generation of a natural 3D image.

[0131] Furthermore, similarly to the first embodiment, the imaging system 100a can be designed flexibly without being restricted by the arrangement of the distance measurement light emitting unit and the light receiving unit.

[0132] 12 is merely an example, and each component of the imaging system 100a may be configured using a device that aggregates multiple components. Also, each functional unit in the image processing device 10a may be distributed and processed using multiple devices.

[0133] <Hardware configuration example>

[0134] Each functional component of the image processing device 10, image processing device 10a, RGB sensor 20, and distance measurement sensor 30 described above may be realized by hardware (e.g., a hardwired electronic circuit, etc.) that realizes each functional component, or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). For example, any process in the present disclosure can be realized by having a CPU (Central Processing Unit) execute a computer program.

[0135] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0136] The present disclosure is not limited to the above-described embodiments and may be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, in the above description, visible light and infrared light are used as the first and second wavelengths of light, but the present disclosure is not limited to this. The imaging system according to the present disclosure may be realized by using light in a wavelength band that is neither visible light nor infrared light as the first or second wavelength of light.

[0137] Furthermore, the above-described first and second embodiments can be combined in any desired manner. For example, if there are multiple shadow areas in one captured image, image processing according to different embodiments may be performed on each shadow area.

[0138] This application claims priority based on Japanese Patent Application No. 2022-063516, filed on April 6, 2022, and Japanese Patent Application No. 2022-063517, the disclosures of which are incorporated herein in their entireties.

[0139] The present disclosure can be used in an imaging system that detects light of different wavelength bands to capture an image.

[0140] 10, 10a Image processing device 11 Visible light image acquisition unit (first image acquisition unit) 12 Infrared light image acquisition unit (second image acquisition unit) 13 Angle of view adjustment unit 14 Conversion unit 15 Difference extraction unit 16 Shadow area detection unit 17 Sampling unit 18 Correction value determination unit 19 Correction unit 20 RGB sensor (first camera) 30 Distance measurement sensor (second camera) 100, 100a Photography system a1 Visible light image a2 Visible light image (grayscaled) b1 Infrared light image b10 Angle of view of infrared light image c1 Image before correction c2 Difference extraction image c3 Shadow area image c4 Corrected image c5 Image before correction c6 Corrected image D1, D2, D14a to D16b Depth value r1 Area p1 Adjacent pixel p2 to p17 Pixel s1 Shadow area s11 Entrance area (first correction area) s12 Middle area (second correction area) s13 Exit area (third correction area) AVG Average value v1 to v3 Correction values

Claims

1. a first image acquisition unit that acquires a first image captured by a first camera that detects light of a first wavelength; a second image acquisition unit that acquires a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure a distance; a difference extraction unit that extracts a difference between the first image and the second image; a shadow region detection unit that detects a shadow region, which is a region where no shadow occurs in the first image and a region where a shadow occurs in the second image, based on the difference; a correction unit that corrects distance information of pixels in the shadow region measured by the distance measuring sensor based on distance information of pixels in the vicinity of the shadow region. Image processing device.

2. The correction unit corrects the distance information of the pixel in the shadow region using the distance information of the pixel adjacent to the shadow region. The image processing device according to claim 1 .

3. the first camera is an RGB camera; A conversion unit converts the RGB information of the first camera into grayscale information, The difference extraction unit extracts the difference using the luminance information. The image processing device according to claim 1 .

4. The light of the first wavelength is visible light, and the light of the second wavelength is infrared light. The image processing device according to any one of claims 1 to 3.

5. The computer a first image acquisition step of acquiring a first image captured by a first camera that detects light of a first wavelength; a second image acquisition step of acquiring a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure a distance; a difference extraction step of extracting a difference between the first image and the second image; a shadow region detection step of detecting a shadow region, which is a region where no shadow occurs in the first image and a region where a shadow occurs in the second image, based on the difference; a correction step of correcting distance information of pixels in the shadow region measured by the distance measuring sensor based on distance information of pixels in the vicinity of the shadow region. Image processing methods.

6. a first image acquisition unit that acquires a first image captured by a first camera that detects light of a first wavelength; a second image acquisition unit that acquires a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure a distance; a difference extraction unit that extracts a difference between the first image and the second image; a shadow region detection unit that detects a shadow region in the second image based on the difference; a correction unit that corrects distance information of the pixels in the shadow area measured by the distance measuring sensor using a correction value based on distance information of a plurality of pixels in the vicinity of the shadow area. Image processing device.

7. a correction value determination unit that determines a correction value based on distance information of a plurality of surrounding pixels around a correction target pixel that is a target for correction by the correction unit; The correction unit corrects distance information of the correction target pixel using the correction value. The image processing device according to claim 6.

8. the shadow region has a first correction region close to a boundary with a vicinity of the shadow region, and a second correction region farther away from the boundary than the first correction region, In the first correction area, the plurality of surrounding pixels are located near the shadow area, The correction value determination unit determines the correction value by a different method when the correction target pixel is in the first correction area and when the correction target pixel is in the second correction area. The image processing device according to claim 7.

9. The light of the first wavelength is visible light, and the light of the second wavelength is infrared light. The image processing device according to any one of claims 6 to 8.

10. The computer a first image acquisition step of acquiring a first image captured by a first camera that detects light of a first wavelength; a second image acquisition step of acquiring a second image captured by a second camera of a distance measurement sensor that detects light of a second wavelength to measure a distance; a difference extraction step of extracting a difference between the first image and the second image; a shadow area detection step of detecting a shadow area in the second image based on the difference; a correction step of correcting distance information of the pixels in the shadow region measured by the distance measuring sensor using a correction value based on distance information of a plurality of pixels in the vicinity of the shadow region. Image processing methods.