Shadow region detection device and shadow region detection method
The shadow area detection device addresses the challenge of correcting shadow-influenced color and distance information in depth camera technologies by using a combination of image acquisition, area division, and polarization filtering to accurately identify and correct shadow areas, enhancing the accuracy and realism of three-dimensional models.
Patent Information
- Application Number
- PCT/JP2024/014390
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-21
AI Technical Summary
Conventional depth camera technologies struggle to accurately correct both color and distance information in shadow areas, leading to discomfort in synthesized three-dimensional models, especially when shadows from environmental light do not match the light source direction in metaverse spaces, and difficulty in determining original subject colors versus shadow-influenced darkened areas.
A shadow area detection device and method that utilizes an image acquisition unit, area dividing unit, polarization filter, and shadow candidate detection unit to differentiate between shadow and non-shadow areas based on polarization images and RGB information, allowing for accurate detection and correction of shadow-influenced color information.
Enables precise detection of shadow areas, improving the accuracy of three-dimensional model generation by distinguishing between shadow and non-shadow regions, thereby reducing the appearance of discomfort caused by shadows and enhancing the realism of virtual models.
Smart Images

Figure JP2024014390_21112024_PF_FP_ABST
Abstract
Description
Shadow area detection device and shadow area detection method
[0001] This application claims priority to Japanese Patent Application No. 2023-081280, filed May 17, 2023, the contents of which are incorporated herein by reference.
[0002] Conventionally, depth cameras (RGBD cameras) have been available that can acquire color information of a subject and distance information to the subject. Based on information acquired using such depth cameras, three-dimensional models such as three-dimensional point cloud data or mesh data are generated. The generated three-dimensional model is then composited into, for example, a metaverse space. In the metaverse space, shadows due to the presence of other three-dimensional models composited in the same metaverse space may be added to the three-dimensional model. When attempting to generate a three-dimensional model of a subject existing in the real world using a depth camera, if shadows due to ambient light at the time of shooting are present, the direction of the shadow may not match the direction of the light source in the metaverse space, resulting in an unnatural appearance. Furthermore, distance information obtained using a depth camera has the drawback of being unable to obtain accurate distance information for images with dark shadow areas. Therefore, to improve the accuracy of distance information measured by a depth camera, a technology has been developed that generates and corrects normal vectors using information from a polarization sensor (see Patent Document 1).
[0003] International Publication No. 2016 / 136086
[0004] However, while the above-described conventional technology may be able to improve the accuracy of distance information, it is unable to correct color information (RGB information). Therefore, shadows are superimposed on the generated three-dimensional model, and when the model is synthesized in the Metaverse space, the darkened areas due to the shadow (hereinafter referred to as "shadow areas") create an unnatural appearance. To reduce the unnatural appearance caused by the shadow, it is conceivable to correct the color information of the shadow areas, which are darker than the surrounding colors. However, even if a part is darker than the surrounding colors, it is difficult to determine whether the part is darker due to the shadow or is the subject's original color, making it difficult to correct the color information of the shadow areas.
[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a shadow area detection device that can detect areas of a subject where the color information is dark due to the influence of a shadow.
[0006] [1] One aspect of this embodiment is a shadow region detection device that includes an image acquisition unit that acquires an image having information about a subject at each coordinate on a two-dimensional plane; a region division unit that divides each coordinate of the acquired image into a plurality of regions based on the information about the subject that each coordinate has; a polarized image acquisition unit that acquires a plurality of polarized images of the subject, each captured through a plurality of polarizing filters having a plurality of different polarization angles; and a shadow candidate detection unit that detects candidate coordinates that are estimated to be shadow areas for each coordinate of the image based on information about the regions of the image divided by the region division unit and the polarized images.
[0007] [2] Furthermore, one aspect of the present embodiment is that in the shadow region detection device described in [1] above, the image acquired by the image acquisition unit includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane, and the region division unit further includes a color region division unit that divides each coordinate of the RGB image into a plurality of regions based on the color information possessed by each coordinate.
[0008] [3] Furthermore, one aspect of this embodiment is the shadow area detection device described in [2] above, further comprising a shadow correction unit that replaces pixel values of coordinates that have been estimated as shadow candidates by the shadow candidate detection unit with values based on pixel values of coordinates that have not been estimated as shadow candidates.
[0009] [4] Furthermore, one aspect of this embodiment is a shadow region detection device according to any one of [1] to [3] above, wherein the image acquired by the image acquisition unit includes a distance image having information relating to the distance to the subject at each coordinate on a two-dimensional plane, and the region division unit further includes a distance region division unit that divides each coordinate of the distance image into a plurality of regions based on the distance information possessed by each coordinate.
[0010] [5] Furthermore, one aspect of this embodiment is a shadow region detection method comprising: an image acquisition step of acquiring an image having information about a subject at each coordinate on a two-dimensional plane; a region division step of dividing each coordinate of the acquired image into a plurality of regions based on the information about the subject held at each coordinate; a polarized image acquisition step of acquiring a plurality of polarized images of the subject, each captured through a plurality of polarizing filters having a plurality of different polarization angles; and a shadow candidate detection step of detecting, for each coordinate of the image, candidates for coordinates estimated to be shadow areas based on information about the image regions divided by the region division step and the acquired polarized images.
[0011] According to this embodiment, it is possible to detect areas of the subject where the color information is dark due to the influence of a shadow.
[0012] 1 is a diagram for explaining shadows detected by the shadow region detection system according to the first embodiment. FIG. 1 is a functional configuration diagram showing an example of the functional configuration of the shadow region detection system according to the first embodiment. FIG. 2 is a diagram for explaining distance region classification processing by a distance region classification unit according to the first embodiment. FIG. 3 is a first flowchart showing an example of distance region classification processing by the distance region classification unit according to the first embodiment. FIG. 4 is a second flowchart showing an example of distance region classification processing by the distance region classification unit according to the first embodiment. FIG. 5 is a diagram for explaining color region classification processing by a color region classification unit according to the first embodiment. FIG. 6 is a first flowchart showing an example of color region classification processing by the color region classification unit according to the first embodiment. FIG. 7 is a second flowchart showing an example of color region classification processing by the color region classification unit according to the first embodiment. FIG. 8 is a diagram for explaining object region classification processing by an object region classification unit according to the first embodiment. FIG. 9 is a diagram for explaining an example of the configuration of a polarization sensor according to the first embodiment. FIG. 10 is an example of a case where color classification is performed based on a polarization angle signal obtained by the polarization sensor according to the first embodiment. FIG. 11 is a diagram showing an example of a SHADOW image according to the first embodiment. FIG. 12 is a functional configuration diagram showing an example of the functional configuration of a shadow region detection system according to the second embodiment. FIG. 13 is a diagram for explaining shadow correction processing by a shadow correction unit according to the second embodiment. Fig. 10 is a functional configuration diagram showing an example of the functional configuration of a shadow correction unit according to a second embodiment. Fig. 11 is a functional configuration diagram showing an example of the functional configuration of an LPF according to the second embodiment. Fig. 12 is a cross-sectional view showing a first example of arrangement of an imaging device including a shadow region detection system according to a third embodiment. Fig. 13 is a cross-sectional view showing a second example of arrangement of an imaging device including a shadow region detection system according to the third embodiment.
[0013] A shadow region detection device according to an aspect of the present invention will be described in detail below with reference to preferred embodiments and the accompanying drawings. Note that the embodiments described below are merely examples, and the embodiments to which the present invention is applicable are not limited to the following embodiments. Furthermore, the phrase "based on XX" in this application means "based on at least XX" and includes cases where the device is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where the device is based on XX after calculation or processing. "XX" is any element (e.g., any information). Furthermore, in the following drawings, the scale and number of elements in each structure may differ from the scale and number of elements in the actual structure to make each configuration easier to understand.
[0014] First Embodiment First, a shadow region detection system 1 according to a first embodiment will be described with reference to FIGS. 1 to 10. FIG.
[0015] 1A and 1B are diagrams illustrating shadows detected by a shadow region detection system according to a first embodiment. FIG. 1A shows an example of capturing an RGB image of an object A as a subject. The figure also shows the angle of view of the captured RGB image. When capturing an RGB image at the angle of view shown in the figure, it is assumed that an object B exists outside the angle of view. In this case, it can be seen that a shadow is projected onto the RGB image of object A due to the influence of a light source such as the sun and object B. The shadow portion is darker (i.e., has lower brightness) than the original color information of object A.
[0016] FIG. 1(B) shows object B, which has a similar shape to object A but different color information. The figure shows the color information that object B originally possesses. In other words, object B has color information as shown, without the influence of shadows. The color information of object A shown in FIG. 1(A) and the color information of object B shown in FIG. 1(B) are identical or similar to each other. In this way, just by looking at an RGB image, it may be difficult to determine whether a part that is darker than the surrounding area (i.e., a part with low brightness) is the original color of the subject, or the color of a shadow cast by an object not visible in the image, as shown in FIG. 1(A).
[0017] When generating a 3D model that is intended to be synthesized into a virtual space such as the Metaverse in the future, if it is possible to determine which parts of the color information of the object being photographed are shadow parts, it is possible to generate the 3D model after correcting the color information. The shadow region detection system 1 according to this embodiment can detect shadow regions from the color information obtained by capturing an image of an object.
[0018] 2 is a functional configuration diagram showing an example of the functional configuration of the shadow area detection system according to the first embodiment. An example of the functional configuration of the shadow area detection system 1 will be described with reference to the diagram. The shadow area detection system 1 includes a shadow area detection device 10, a storage device 21, a storage device 22, a distance measurement sensor 31, an RGB sensor 32, and a polarization sensor 33.
[0019] The storage device 21 stores parameters and the like required for processing by the shadow area detection device 10. The parameters and the like stored in the storage device 21 may be rewritten as appropriate by a user or the like, or may change depending on environmental information such as the time of day, temperature, or weather. The storage device 22 stores information for identifying the shadow area detected by the shadow area detection device 10. Note that both the storage device 21 and the storage device 22 may be included in the shadow area detection device 10.
[0020] The distance measurement sensor 31 acquires three-dimensional information of the subject. Specifically, the distance measurement sensor 31 acquires a distance image by measuring the distance to the subject at multiple points. The distance measurement sensor 31 may use, for example, a ToF (Time of Flight) method to two-dimensionally irradiate the subject with light and measure the distance based on the time it takes to receive the reflected light. The distance measurement sensor 31 outputs the acquired distance image to the shadow area detection device 10.
[0021] The RGB sensor 32 captures an RGB image of the same subject as the subject being measured by the distance measurement sensor 31. It is preferable that the coordinates of the distance image acquired by the distance measurement sensor 31 correspond to the coordinates of the RGB image acquired by the RGB sensor 32. The distance measurement sensor 31 and the RGB sensor 32 may be components of a depth camera (RGBD camera) (not shown). The RGB sensor 32 includes, for example, multiple photodiodes with filters for each of the RGB colors arranged in a Bayer pattern, and acquires luminance information for each of the RGB colors. The RGB sensor 32 outputs the acquired RGB image to the shadow area detection device 10, for example, as a RAW image.
[0022] The polarization sensor 33 acquires a polarized image of the same subject that is the subject of distance measurement by the distance measurement sensor 31 and the subject of image capture by the RGB sensor 32. The polarization sensor 33 captures a plurality of polarized images that are captured through a plurality of polarizing filters (polarizers) with different polarization angles. The polarization sensor 33 outputs the acquired polarized images to the shadow area detection device 10.
[0023] The shadow region detection device 10 includes an image acquisition unit 111, a resizing unit 113, a hue / luminance calculation unit 115, a region division unit 119, a polarized image acquisition unit 131, a polarization angle calculation unit 132, a resizing unit 133, and a shadow candidate detection unit 150. Each of these functional units is implemented using, for example, electronic circuits. Each functional unit may also include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as needed. Each function may also be implemented by a computer having a central processing unit (CPU) and software.
[0024] The image acquisition unit 111 acquires an image having information about a subject at each coordinate on a two-dimensional plane. The image includes an RGB image having information about the color of the subject at each coordinate on the two-dimensional plane, and a distance image having information about the distance to the subject at each coordinate on the two-dimensional plane. That is, the image acquisition unit 111 acquires at least one of an RGB image or a distance image. Specifically, the image acquisition unit 111 acquires an RGB image from the RGB sensor 32. The image acquisition unit 111 performs de-Bayer (demosaic) processing on the acquired RGB image to obtain image signals for each RGB color. The image acquisition unit 111 also acquires a distance image from the ranging sensor 31.
[0025] The resizing unit 113 acquires information about the distance image from the image acquisition unit 111. If the resolution of the distance image differs from the resolution of each RGB color obtained from the RGB image, the resizing unit 113 resizes (upconverts) the distance image to match the resolution. Note that the resizing unit 113 may also acquire image signals for each RGB color from the image acquisition unit 111 and resize (downconvert) the image signals for each RGB color to match the resolution.
[0026] The region dividing unit 119 acquires a distance image and an RGB image that have been adjusted to the same resolution by the resizing unit 113. The region dividing unit 119 divides each coordinate of the acquired image into multiple regions based on information about the subject at each coordinate. Specifically, the region dividing unit 119 includes a distance region dividing unit 116, a color region dividing unit 117, and an object region dividing unit 118. The distance region dividing unit 116 performs region dividing processing based on the distance image. It can also be said that the distance region dividing unit 116 divides each coordinate of the distance image into multiple regions based on distance information at each coordinate. The color region dividing unit 117 performs region dividing processing based on the RGB image. It can also be said that the color region dividing unit 117 divides each coordinate of the RGB image into multiple regions based on color information at each coordinate. The object region dividing unit 118 performs region dividing processing based on the region dividing results performed by the distance region dividing unit 116 and the region dividing results performed by the color region dividing unit 117. The region dividing processing is a process of dividing each coordinate of the image into regions for each of multiple objects present in the image. In the region dividing processing, the same identification number (INDEX number) is assigned to regions of similar colors within the same object.
[0027] First, we will explain the region division process performed by distance region division unit 116. Distance region division unit 116 acquires a distance image that has been resized by resizing unit 113 and adjusted to the same resolution as the image signals of each color of RGB. Distance region division unit 116 also acquires parameters DTH1max, DTH1min, DTH2, and DTH3 used in the distance region division process from storage device 21.
[0028] DTH1max sets the distance for removing distance data of the subject's background from the shadow detection area. In other words, DTH1max can be considered the maximum value in the depth direction of the subject. DTH1min sets the distance for removing distance data of the subject's front side from the shadow detection area. In other words, DTH1min can be considered the maximum value in the front direction of the subject. DTH2 is a threshold for determining whether distance values of adjacent coordinates are on the same plane. DTH3 is a threshold for removing abnormal values due to large random noise caused by the sensor, dead pixels, etc. The distance area division unit 116 uses these parameters to divide the object area of the subject and assigns a distance INDEX number to each area. This is based on the idea that if the object is the same, the distance values of adjacent pixels will not be significantly different. The distance INDEX number is a number assigned to each pixel based on distance information, and the same number is assigned to pixels that are estimated to represent the same object.
[0029] The operation of assigning distance INDEX numbers by the distance area dividing unit 116 will be described in detail with reference to FIGS. 3, 4A and 4B.
[0030] FIG. 3 is a diagram illustrating the distance area division process performed by the distance area division unit according to the first embodiment. For ease of explanation, the figure shows a distance image (depth data image) acquired by the image acquisition unit 111, represented using a small number of pixels. In the figure, the values of the distance image (distance values) are represented by shades of gray. Specifically, darker areas indicate closer objects, and lighter areas indicate farther objects. FIG. 3A shows coordinates to which numbers are assigned to identify each coordinate for illustrative purposes. FIG. 3B shows distance INDEX numbers assigned by the distance area division unit 116.
[0031] 4A and 4B (hereinafter sometimes simply referred to as FIG. 4) are flowcharts showing an example of distance region division processing by the distance region division unit according to the first embodiment. The illustrated flowchart shows the flow of processing when sequential processing is performed by scanning horizontally and vertically from the upper left pixel of the distance image shown in FIG. 3. The flowchart shown in FIG. 4 loops so as to perform processing by sequential scanning as indicated by the arrows shown in FIG. 3.
[0032] First, the distance area division unit 116 acquires distance values (depth data) for a 3×3 pixel area from the distance screen (step S111). The acquired 3×3 pixels may contain data that is different from the surrounding pixels due to sensor noise or dead pixels. The distance area division unit 116 first assigns a distance INDEX number to such data in steps S112 and S113. Specifically, the distance area division unit 116 compares the center pixel of the 3×3 pixels with all surrounding pixels to determine whether the difference in distance value is greater than or equal to +DTH3 or less than or equal to −DTH3 (step S112). If the distance area division unit 116 determines that there is a predetermined difference in distance value (i.e., step S112; YES), it sets the distance INDEX number of the center pixel of the 3×3 pixels to the distance INDEX number of the pixel to the left (step S113). If a distance INDEX number has not yet been assigned to the top left 3×3 area, the distance INDEX number is set to 0. If the distance area dividing unit 116 determines that there is no predetermined difference in the distance values (i.e., step S112; NO), the process proceeds to step S114.
[0033] Looking at FIG. 3A, there is a coordinate labeled F in the center. Assume that the difference in distance value between the pixel at that coordinate and the surrounding pixels is ±DTH3. In this case, the pixel at that coordinate becomes the distance INDEX number of the pixel to the left. Specifically, as shown in FIG. 3B, the distance INDEX number of the pixel to the left is determined to be 1, just like the distance INDEX number of the pixel to the left. The determined distance INDEX number is stored in a temporary storage unit (not shown), and the process moves to the next pixel and loops back to the beginning of the flow. Note that in the process described below, a distance INDEX number is assigned to each pixel, and the assigned distance INDEX numbers are stored in a temporary storage unit.
[0034] If step S112 is NO, the distance area dividing unit 116 then performs processing in steps S114 and S115 to exclude background objects behind the subject and unnecessary objects in front of the subject from the shadow detection targets. Specifically, the distance area dividing unit 116 performs a determination using distance thresholds DTH1max and DTH1min. DTH1max is a threshold for excluding objects farther away from the subject from the shadow detection targets. DTH1min is a threshold for excluding objects closer to the subject from the shadow detection targets. First, the distance area dividing unit 116 determines whether the distance value of the center pixel of the 3x3 pixel array is equal to or less than DTH1min or equal to or greater than DTH1max (step S114). If the distance value exceeds the threshold (i.e., step S114; YES), the distance area dividing unit 116 sets the distance INDEX number of the center pixel to 0 (step S115). Furthermore, if the distance value does not exceed the threshold (i.e., step S114; NO), the distance area dividing unit 116 proceeds to step S116. Specifically, in the example shown in FIG. 3A, coordinates without numbers indicate areas exceeding DTH1max, i.e., areas determined to be background behind the subject. Furthermore, areas numbered 33 to 39 indicate areas below DTH1min, i.e., areas determined to be extraneous objects in front of the subject. Such areas are assigned a distance INDEX number of 0, as shown in FIG. 3B.
[0035] If step S114 is NO, the distance area dividing unit 116 then assigns a distance INDEX number to the newly detected object in steps S116 and S117. In other words, the distance area dividing unit 116 can be said to assign a new distance INDEX number to the contour portion of the object. Specifically, since the distance area dividing unit 116 sequentially scans horizontally starting from the upper left pixel, when scanning a certain pixel, it can be said that the top three pixels and the left pixel of the 3x3 pixel array have already been scanned. The distance area dividing unit 116 determines whether any of the scanned pixels still have a distance INDEX number other than 0 (step S116). If no distance INDEX number other than 0 still exists (i.e., step S116; YES), the distance area dividing unit 116 assigns a distance INDEX number to the pixel (step S117). Furthermore, if there is still a distance INDEX number other than 0 (i.e., step S116; NO), the distance area dividing unit 116 proceeds to step S118. By performing this operation, the distance area dividing unit 116 can assign distance INDEX numbers to the central pixels of the 3x3 pixel array that have not previously been assigned distance INDEX numbers. In the illustrated example, the pixel assigned 1 in FIG. 3A is the first pixel to be determined to contain an object when scanning from the top left, and therefore is assigned a distance INDEX number of 1. Furthermore, the pixel assigned 22 is assigned a distance INDEX number of 4 because it was assigned after distance INDEX numbers up to 3 had already been assigned.
[0036] If step S116 is NO, the distance area dividing unit 116 then determines in steps S118 to S121 whether the portions determined to be objects are on the same plane. That is, in steps S118 to S121, the distance area dividing unit 116 determines whether to assign the same distance INDEX number or different distance INDEX numbers. Here, the processing of step S118 is performed when, among the 3×3 pixels, the top three pixels and the left pixel that have already been scanned and assigned distance INDEX numbers have distance INDEX numbers other than 0. In such a case, the distance area dividing unit 116 determines whether the center pixel of the 3×3 pixels is on the same plane as the pixel already assigned a distance INDEX number, based on the DTH2 threshold level.
[0037] Specifically, the distance area division unit 116 determines whether any of the distance values of the top three pixels and the left pixel is within −DTH2 to +DTH2 of the distance value of the central pixel of the 3 × 3 array (step S118). If any of the distance values of the top three pixels and the left pixel is within −DTH2 to +DTH2 of the distance value of the central pixel of the 3 × 3 array (i.e., step S118; YES), the distance area division unit 116 regards them as belonging to the same plane and assigns them the same distance INDEX number. In this case, if there are multiple pixels whose distance values are within −DTH2 to +DTH2 of the central pixel, all of those pixels are assigned the smallest distance INDEX number among them (step S119). Furthermore, if distance INDEX numbers have already been assigned to the top three pixels and the left one pixel and the distance INDEX numbers are rewritten, the coordinates scanned up to that point and for which distance INDEX numbers have been stored are referenced, and all coordinates having the same distance INDEX number are rewritten (step S121).
[0038] In the illustrated example, the pixels with coordinates 2, 6, 12, and 13 in FIG. 3A were assigned distance INDEX numbers of 2 until scanning processing was performed on the pixel with coordinate 18. However, when determining the pixel with coordinate 18, the distance INDEX number was reassigned to 1. This is because the distance value of the pixel with coordinate 17 was within the range of -DTH2 to +DTH2. At that time, as shown in FIG. 3B, the distance INDEX numbers of the pixels with coordinates 2, 6, 12, and 13 outside the 3×3 pixel area that had already been stored as distance INDEX numbers of 2 were also reassigned to 1. After this processing, processing moves on to the next pixel.
[0039] If any of the distance values of the top three pixels and the left pixel is not within the range of -DTH2 to +DTH2 of the distance value of the center pixel of the 3x3 matrix (i.e., step S118; NO), the distance area dividing unit 116 updates the distance INDEX number of the target pixel to a number other than the previous distance INDEX number, and sets this to a new distance INDEX number (step S120). For example, for the pixel with coordinate 7 shown in FIG. 3A, the distance INDEX number is 2 during the scanning process of this pixel. However, if the distance value between the left pixel (i.e., the pixel with coordinate 6) and the upper left pixel (i.e., the pixel with coordinate 2) is not within the range of -DTH2 to +DTH2, the distance INDEX number is updated and stored as 3 (step S120). After performing this process, the distance area dividing unit 116 proceeds to processing the next pixel (step S122).
[0040] The distance area division unit 116 scans all pixels in the above manner. That is, in this embodiment, the distance INDEX number is updated as a result of multiple determinations. Figure 3B shows the final results obtained for all pixels.
[0041] Returning to FIG. 2, the distance region dividing unit 116 outputs information about the finally obtained distance INDEX number to the object region dividing unit 118 .
[0042] Next, the region division process performed by the color region division unit 117 will be described. The color region division unit 117 acquires RGB color signals based on the RGB image acquired by the image acquisition unit 111, and a distance image resized by the resizing unit 113 to have the same resolution as the RGB image signals. The color region division unit 117 also acquires WTH, DTH1min, DTH1max, HUETH, and SATTH, which are parameters used in the color region division process. DTH1min and DTH1max may be the same parameters as those used in the distance region division process performed by the distance region division unit 116.
[0043] WTH is a threshold for excluding abnormal values due to prominent noise from the RGB sensor, dead pixels, etc. WTH may be a value that varies depending on temperature. HUETH is a threshold for hue for dividing regions by color. SATTH is a threshold for color saturation. The color region dividing unit 117 uses these parameters to divide object regions of the subject and assigns a color INDEX number to each region. This is based on the idea that if the same color is used on the same object, even if it is a shadow and a non-shadow area, both will have similar hues. The color INDEX number is a number assigned to each pixel based on hue information, and the same number is assigned to pixels estimated to be of similar colors within the same object. The distance INDEX number and the color INDEX number are different approaches for dividing the same object into regions based on distance and regions based on color, respectively.
[0044] In cases where the hue cannot be determined, for example, for colorless black, gray, or white objects, the color saturation is low, at the noise level. Therefore, in this embodiment, objects below a certain color saturation level are considered to be the same black, white, and gray objects.
[0045] The operation of assigning color INDEX numbers by the color region dividing unit 117 will be described in detail with reference to FIGS. 5, 6A and 6B.
[0046] FIG. 5 is a diagram for explaining the color region segmentation process performed by the color region segmentation unit according to the first embodiment. For ease of explanation, the figure shows an RGB image acquired by the image acquisition unit 111, represented using a small number of pixels. In the figure, pixel values are represented by shades of gray. In FIG. 5A, each coordinate is assigned a number to identify the coordinate used for explanation. FIG. 5B shows the color INDEX number assigned by the color region segmentation unit 117.
[0047] 6A and 6B (hereinafter, sometimes simply referred to as FIG. 6) are flowcharts showing an example of color gamut segmentation processing by the color gamut segmentation unit according to the first embodiment. The illustrated flowchart shows the flow of processing when sequential processing is performed by scanning horizontally and vertically from the upper left pixel of the RGB image shown in FIG. 5. The flowchart shown in FIG. 6 loops so that processing is performed by sequential scanning as indicated by the arrows in FIG. 5.
[0048] First, the color region dividing unit 117 converts the input RGB color signals into luminance Y, hue HUE, and color saturation SAT. For this conversion, for example, the following mathematical expressions (1) to (5) may be used.
[0049] Y=0.2126×R+0.7152×G+0.0722×B...(1) Pb=-0.1146×R-0.3854×G+0.5000×B...(2) Pr=0.5000×R-0.4542×G-0.0458×B...(3) HUE=ATAN(Pr / Pb)...(4) SAT=SQRT(PB^2+Pr^2)...(5)
[0050] The color region dividing unit 117 calculates HUE, SAT, and Y for all 3 × 3 pixels using equations (1) to (5) (step S211). The color region dividing unit 117 also obtains the distance value (depth value) of the center pixel of the 3 × 3 pixels.
[0051] Next, in steps S212 and S213, the color region segmentation unit 117 processes pixels with abnormal values due to RGB sensor noise, dead pixels, and pixels with pixel defects known as white or black defects. Specifically, the color region segmentation unit 117 first compares the RGB signals of the center pixel of the 3×3 pixel array with the RGB signals of the surrounding eight pixels of the 3×3 pixel array (step S212). If any of the surrounding eight pixels has a pixel value greater than or equal to +WTH or less than −WTH of the center pixel (i.e., step S212; YES), the color region segmentation unit 117 considers the center pixel to be a dead pixel and assigns it the color INDEX number of the pixel to the neighboring pixel on the left (step S213). Note that if a color INDEX number has not yet been assigned to the top-left 3×3 pixel array, the color INDEX number is set to 0.
[0052] In the illustrated example, in FIG. 5A, the pixel designated W has an RGB value that is further away from the surrounding pixels by ±WTH. In this case, the color region dividing unit 117 sets the color INDEX number of the pixel designated W to the color INDEX number of the neighboring pixel. Looking at FIG. 5B, the color INDEX number of the pixel designated W in FIG. 5A is the same as the color INDEX number of the neighboring pixel (i.e., the pixel with coordinate 16). The determined color INDEX number is stored in a temporary storage unit (not shown), and the process moves to the next pixel and loops back to the beginning of the flow. Note that in the process described below, a color INDEX number is assigned to each pixel, and the assigned color INDEX number is stored in the temporary storage unit.
[0053] If step S212 is NO, then in steps S214 and S215, the color region dividing unit 117 performs processing to exclude background behind the subject and extraneous objects in front of the subject from the shadow detection targets. Specifically, the color region dividing unit 117 performs a determination using distance thresholds DTH1max and DTH1min. First, the color region dividing unit 117 determines whether the distance value of the center pixel is equal to or less than DTH1min or equal to or greater than DTH1max (step S214). If the distance value exceeds the threshold (i.e., step S214; YES), the color region dividing unit 117 sets the color INDEX number of the center pixel to 0 (step S215). If the distance value does not exceed the threshold (i.e., step S214; NO), the color region dividing unit 117 proceeds to step S216. Specifically, in the example shown in Figure 5(A), coordinates without numbers indicate areas exceeding DTH1max, i.e., areas determined to be background behind the subject. Also, areas numbered 35 through 41 indicate areas below DTH1min, i.e., areas determined to be extraneous objects in front of the subject. As shown in Figure 5(B), such areas are assigned a color INDEX number of 0.
[0054] If step S214 is NO, the color region division unit 117 then assigns a color INDEX number to the newly detected object in steps S216 and S217. In other words, it can be said that new color INDEX numbers are assigned to the contours of the object. Specifically, the color region division unit 117 sequentially scans horizontally starting from the upper left pixel. Therefore, when scanning a certain pixel, the top three pixels and the left pixel of the 3x3 pixel set have already been scanned. The color region division unit 117 determines whether any of the scanned pixels still have a color INDEX number other than 0 (step S216). If no color INDEX number other than 0 still exists (i.e., step S216; YES), the color region division unit 117 assigns a color INDEX number to the pixel (step S217). Furthermore, if there is still a distance INDEX number other than 0 (i.e., step S216; NO), the color region classification unit 117 proceeds to step S218. By performing this process, the color region classification unit 117 can assign color INDEX numbers to the central pixels of the 3x3 pixel array that have not previously been assigned a color INDEX number. In the illustrated example, the pixel assigned 1 in FIG. 5A is the first pixel to be determined to contain an object when scanning from the top left, and therefore is assigned a distance INDEX number of 1. Furthermore, the pixel assigned 24 is assigned a distance INDEX number of 4 because it was assigned after distance INDEX numbers up to 3 had already been assigned.
[0055] If step S216 is NO, the color region division unit 117 then determines in steps S218 and S219 whether the pixel values are in the black, white, or gray range. The SATTH threshold is used to determine whether the pixel values are in the black, white, or gray range. Specifically, the color region division unit 117 determines whether there is at least one pixel in either the top three pixels or the left pixel that has a color INDEX number of 1 or more, and whether the SAT (see equation (5) above) of the top three pixels and the left pixel is within SATTH (step S218). If even one pixel meets this condition (i.e., step S218; YES), these pixels are considered to be of the same color and are assigned the same color INDEX number. At this time, if there are multiple pixels within SATTH, all of those pixels are assigned the smallest distance INDEX number among them (step S219). In the illustrated example, the pixels marked with 7 and 8 in FIG. 5A had a color INDEX number of 4 until they were scanned. However, when scanning is performed on the pixel marked with 8, the SAT value of the pixel to the left marked with 7 is within SATTH, so the color INDEX number is reassigned to 2. If there is already a pixel stored outside the 3×3 region that has a color INDEX number of 4, the color INDEX number is reassigned to 2 and restored. The color region classification unit 117 performs this process before moving on to the next pixel.
[0056] If step S218 is NO, then the color region division unit 117 determines whether the pixels have the same hue in steps S220 through S223. That is, in steps S220 through S223, the color region division unit 117 determines whether to assign the same color INDEX number or different color INDEX numbers. Here, the processing of step S220 is performed when, among the 3×3 pixels, the top three pixels and the left pixel that have already been scanned and assigned color INDEX numbers have color INDEX numbers other than 0. In such a case, the color region division unit 117 determines whether the center pixel of the 3×3 pixels has the same hue as the pixel already assigned a color INDEX number based on the HUETH threshold level.
[0057] Specifically, the color region division unit 117 determines whether any of the hue values of the top three pixels and the left pixel is within −HUETH to +HUETH of the hue value of the center pixel of the 3×3 matrix (step S220). If any of the hue values of the top three pixels and the left pixel is within −HUETH to +HUETH of the hue value of the center pixel of the 3×3 matrix (i.e., step S220; YES), the color region division unit 117 considers the hue values of the top three pixels and the left pixel to be the same hue and assigns the same color INDEX number to them. In this case, if there are multiple pixels whose hue is within −HUETH to +HUETH of the hue of the center pixel, all of those pixels are assigned the smallest color INDEX number among them (step S221). Furthermore, if color INDEX numbers have already been assigned to the top three pixels and the left one pixel and the color INDEX numbers are being rewritten, the coordinates where color INDEX numbers have been stored through scanning up to that point are referenced, and all coordinates having the same color INDEX number are rewritten (step S223).
[0058] In the illustrated example, the pixels at coordinates 6 and 14 in FIG. 5A were assigned a color INDEX number of 3 until scanning was performed on the pixel at coordinate 20. However, when determining the pixel at coordinate 20, the color INDEX number is reassigned to 1. This is because the HUE value of the pixel at coordinate 19 was within the range of −HUETH to +HUETH. At the same time, as shown in FIG. 5B, the color INDEX numbers of the pixels at coordinates 6 and 14 outside the 3×3 pixel area that had already been stored as color INDEX numbers 3 are also reassigned to 1. After this processing, processing moves on to the next pixel (step S224).
[0059] If the hue values of any of the top three pixels and the left pixel are not within the range of -HUETH to +HUETH of the hue value of the center pixel of the 3x3 matrix (i.e., step S220; NO), the color region division unit 117 updates the color INDEX number of the target pixel to a number other than the color INDEX number it has been assigned so far, and sets the new color INDEX number (step S222). For example, for a pixel with coordinate 34 in FIG. 5A, the color INDEX number is 1 during the scanning process of this pixel. However, if the hue values of the left pixel (i.e., the pixel with coordinate 33), the upper left pixel (i.e., the pixel with coordinate 28), and the pixel above (i.e., the pixel with coordinate 29) are not within the range of -HUETH to +HUETH, the color INDEX number is updated and stored as 5 (step S222). After the above process is completed, the distance region dividing unit 116 moves on to process the next pixel (step S224).
[0060] The color region classification unit 117 scans all pixels in the above manner. That is, in this embodiment, the color INDEX numbers are updated as a result of multiple determinations. Figure 5B shows the final results obtained for all pixels.
[0061] Returning to FIG. 2, the color region dividing unit 117 outputs information about the finally obtained color INDEX number to the object region dividing unit 118 .
[0062] The object region dividing unit 118 acquires information about the distance INDEX number from the distance region dividing unit 116, and acquires information about the color INDEX number from the color region dividing unit 117. The object region dividing unit 118 performs final region division of the object based on the acquired information.
[0063] FIG. 7 is a diagram illustrating the object region division process performed by the object region division unit according to the first embodiment. An example of the object region division process performed by the object region division unit 118 will be described with reference to the same figure. The object region division unit 118 considers the color INDEX number as a reference and assigns an OBJ_INDEX number to pixel portions having a color INDEX number other than 0. The reason the object region division unit 118 uses the color INDEX number as a reference (rather than the distance INDEX number) is that the RGB sensor 32 is less susceptible to noise (compared to the distance measurement sensor 31) and the boundaries of objects are relatively smooth. Note that this embodiment is not limited to this example, and the object region division unit 118 may also use the distance INDEX number as a reference.
[0064] The object region division unit 118 first assigns OBJ_INDEX numbers to pixels whose color INDEX number is other than 0, with the color INDEX number being the higher order and the distance INDEX number being the lower order. For example, if the color INDEX number is 2 and the distance INDEX number is 3, the OBJ_INDEX number will be 2_3. In this manner, OBJ_INDEX numbers are assigned to all pixels. The result of assigning OBJ_INDEX numbers to all pixels is shown in FIG. 7A.
[0065] Next, the object region division unit 118 scans the OBJ_INDEX numbers for each 3×3 pixel region. If the OBJ_INDEX number at the center of the 3×3 pixel region has a non-zero leading digit and a zero leading digit, the object region division unit 118 assigns the number with the largest number of surrounding 3×3 pixels having the same leading digit and the same lowest digit, and updates the OBJ_INDEX number to that number. If this condition does not apply to the surrounding pixels, the number is set to 0_0 and the OBJ_INDEX number is updated. Note that a pixel that meets this condition corresponds to the pixel with the OBJ_INDEX number 2_0 in FIG. 7A. This pixel is updated to 2_3, as shown in FIG. 7B. Note that if the OBJ_INDEX number at the center of the 3×3 pixel region does not meet the condition that the leading digit is non-zero and the lowest digit is zero, the OBJ_INDEX number is not updated. In this way, the OBJ_INDEX number for the final object area classification is obtained.
[0066] 2 , the object region division unit 118 outputs the OBJ_INDEX number resulting from the final object region division to the shadow candidate 1 detection unit 151 and the shadow candidate 2 detection unit 152. In the following description, a configuration including the shadow candidate 1 detection unit 151 and the shadow candidate 2 detection unit 152 may be referred to as the shadow candidate detection unit 150. It can also be said that the shadow candidate detection unit 150 detects candidates for coordinates estimated to be shadow areas for each coordinate in the image, based on information about the image regions divided by the region division unit 119 and the polarized image acquired by the polarized image acquisition unit 131.
[0067] The shadow candidate 1 detection unit 151 then detects a candidate that is estimated to be a shadow (hereinafter referred to as a shadow candidate) from among a group of pixels assigned the same OBJ_INDEX number. The shadow candidate 1 detection unit 151 obtains the OBJ_INDEX number from the object region classification unit 118 and obtains YTH from the storage device 21. Here, YTH is a threshold value for determining whether or not there is a difference between the RGB luminance signal Y and the darkness of the shadow. YTH may be a value that varies depending on the time when the image to be processed was captured (i.e., the time when the image acquisition unit 111 acquires the image), the weather, etc.
[0068] The shadow candidate 1 detection unit 151 first scans the OBJ_INDEX numbers starting from the top left and detects how many pixels have the same OBJ_INDEX number for each number. If there are five or fewer pixels with the same OBJ_INDEX number, the shadow candidate 1 detection unit 151 considers the object to be too small to have a shadow. In this case, the shadow candidate 1 detection unit 151 sets the value of the output signal, shadow candidate 1, to 0. If there are six or more pixels with the same OBJ_INDEX number, the shadow candidate 1 detection unit 151 calculates the average of the three highest brightness values of the six pixels in order of brightness Y. If there is a pixel with a brightness value YTH lower than the calculated average brightness value, the shadow candidate 1 detection unit 151 considers the pixel to be in a shadow area and sets the value of the output signal, shadow candidate 1, to 1. If there are no pixels with a YTH lower brightness value among the calculated average brightness values, the shadow candidate 1 detection unit 151 determines that no shadow exists and sets shadow candidate 1 to 0. The shadow candidate 1 detection unit 151 outputs the shadow candidate 1 obtained in this way to the shadow candidate 2 detection unit 152. Note that the shadow candidate 1 output signal is a value that exists for each OBJ_INDEX number.
[0069] Next, we will explain the functions of the polarization image acquisition unit 131, the polarization angle calculation unit 132, and the resizing unit 133. As a prerequisite for the functions of the polarization angle calculation unit 132 and the resizing unit 133, we will explain an example of the configuration of the polarization sensor 33.
[0070] 8A and 8B are diagrams illustrating an example of the configuration of the polarization sensor according to the first embodiment. As shown in FIG. 8A, the polarization sensor 33 includes a microlens array 331, a polarizing plate array 332, and a pixel array 333. The microlens array 331 includes a plurality of microlenses, the polarizing plate array 332 includes a plurality of polarizers (polarizing plates), and the pixel array 333 includes a plurality of pixels (picture elements). As shown in the figure, the polarizers are mounted so that the polarization angle changes by 45 degrees for each pixel. FIG. 8B shows the polarization angle for each polarizer. As shown in the figure, the polarization sensor 33 can detect polarization degrees of 0 degrees, 45 degrees, 90 degrees, and 135 degrees using four pixels.
[0071] Returning to FIG. 2 , the polarization image acquisition unit 131 acquires a polarization image captured by the polarization sensor 33. The polarization image includes signals in four polarization directions. The polarization image can also be said to be an image of a subject captured through polarization filters with a plurality of different polarization angles. It is widely known that the polarization angle can be calculated from the four polarization direction signals obtained from the polarization sensor 33, and this can be done, for example, by calculation using the following equations (6) to (8). The signals of the four polarization angles obtained from the polarization sensor 33 will be referred to as H0, H45, H90, and H135.
[0072] Q=H0-H90...(6) U=H45-H135...(7)
[0073] In this case, the polarization angle AoLP can be calculated by the following equation (8).
[0074] AoLP=1 / 2ATAN(U / Q)...(8)
[0075] The polarization angle calculation unit 132 obtains the AoLP by performing calculations using the above-mentioned equations (6) to (8).
[0076] The resizing unit 133 performs resizing so that the number of pixels with the polarization angle is the same as that of the RGB sensor 32 .
[0077] Here, we note that the polarization angle of the light and shadow areas of a subject can be very different, even for the same object. Shadows are created when light hits them in different ways. Therefore, the polarization angle of the shadow areas will be different from the polarization angle of the light-hit areas.
[0078] 9A and 9B show an example of a case where an image is color-coded based on the polarization angle signals obtained by the polarization sensor according to the first embodiment. FIG. 9A shows the signals for the four polarization angles obtained by the polarization sensor. FIG. 9B shows an example of a case where the polarization angle AoLP is calculated from these signals using Equation (8) and the angle is color-coded. As shown in the figure, it can be seen that the polarization angles of the shadowed and non-shadowed portions are different even for the same object.
[0079] Returning to FIG. 2, the process of detecting the final shadow candidate based on the shadow candidate 1 and the information obtained by the polarization sensor 33 will be described.
[0080] The shadow candidate 2 detection unit 152 acquires shadow candidate 1 from the shadow candidate 1 detection unit 151 and acquires the OBJ_INDEX number from the object area classification unit 118. The shadow candidate 2 detection unit 152 also acquires DEGTH from the storage device 21. DEGTH is a threshold used to determine whether there is a difference in polarization angle. The shadow candidate 2 detection unit 152 calculates the average polarization angle AoLP for portions of the same OBJ_INDEX number where the shadow candidate 1 signal is 1. Specifically, the shadow candidate 2 detection unit 152 may extract the polarization angle AoLP of the portion corresponding to the shadow candidate 1 signal being 1 from the information obtained after the polarization angle AoLP is calculated by the polarization angle calculation unit 132 and resizing is performed by the resizing unit 133. The shadow candidate 2 detection unit 152 stores the calculated polarization angle AoLP as AoLP_Shadow1. Next, the shadow candidate 2 detection unit 152 calculates the average polarization angle AoLP of the portion of the same OBJ_INDEX number where the signal of the shadow candidate 1 is 0. The shadow candidate 2 detection unit 152 sets the calculated polarization angle AoLP as AoLP_Shadow0.
[0081] Next, the shadow candidate 2 detection unit 152 calculates the difference in angle between the calculated AoLP_Shadow1 and AoLP_Shadow0. The shadow candidate 2 detection unit 152 also calculates whether the calculated difference is equal to or greater than DEGTH. The shadow candidate 2 detection unit 152 determines that a portion where the calculated difference is equal to or greater than DEGTH is the final shadow portion, and outputs a SHADOW signal of 1. If the calculated difference is equal to or less than DEGTH, the shadow candidate 2 detection unit 152 sets the SHADOW signal to 0. This SHADOW signal exists for each pixel and corresponds to the resolution of the RGB image and the distance image. In other words, information including a SHADOW signal for each pixel can also be referred to as a SHADOW image.
[0082] 10 is a diagram showing an example of a shadow image according to the first embodiment. The shadow image shown corresponds to FIGS. 3, 5, and 7. 1 indicates a shadow area, and 0 indicates a non-shadow area. According to this embodiment, by referring to a shadow image as shown in the figure, it becomes possible to determine whether or not each pixel in the RGB image and the distance image is a shadow.
[0083] It is preferable that the shadow image be stored in association with the RGB image and the distance image. By storing the shadow signal in association with the RGB image and the distance image, it becomes easy to determine which part is the shadow of the original subject by referring to the shadow image, even in cases where a three-dimensional model is formed.
[0084] The RGB image, distance image, and shadow image according to this embodiment may be captured as continuous frames and treated as a moving image.
[0085] Summary of Embodiment 1 According to the embodiment described above, the shadow region detection device 10 includes the image acquisition unit 111 to acquire an image having subject information at each coordinate on a two-dimensional plane; the region division unit 119 to divide each coordinate of the acquired image into multiple regions based on the subject information at each coordinate; the polarized image acquisition unit 131 to acquire multiple polarized images of the subject captured through polarizing filters having multiple different polarization angles; and the shadow candidate detection unit 150 to detect coordinate candidates estimated to be shadow areas for each coordinate of the image based on information about the image regions divided by the region division unit 119 and the polarized image acquired by the polarized image acquisition unit 131. That is, according to this embodiment, it is possible to detect areas of the subject where color information is darkened due to the influence of shadows. Since this embodiment can detect areas of the subject where color information is darkened due to the influence of shadows, when attempting to generate a three-dimensional model, it is possible to generate a three-dimensional model with reduced influence of shadows by correcting the color information of the detected shadow areas.
[0086] Furthermore, according to the above-described embodiment, the image acquired by the image acquisition unit 111 includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane, and the region division unit 119 divides each coordinate of the RGB image into multiple regions based on the color information possessed by each coordinate. That is, according to this embodiment, shadow candidates are detected based on a high-resolution RGB image and a polarized image. Therefore, according to this embodiment, shadow candidates are detected based on a high-resolution RGB image, and therefore shadow candidates can be detected with high accuracy.
[0087] Furthermore, according to the above-described embodiment, the image acquired by the image acquisition unit 111 includes a distance image having information about the distance to the subject at each coordinate on a two-dimensional plane, and the region division unit 119 divides each coordinate of the distance image into multiple regions based on the distance information held by each coordinate. That is, according to this embodiment, shadow candidates are detected based on the distance image and the polarization image. The distance image can be used to accurately determine whether or not the objects are the same. Therefore, according to this embodiment, shadow candidates can be detected with high accuracy.
[0088] [Second and Third Embodiments] Next, a shadow region detection system 1A according to a second and third embodiment will be described with reference to Figs. 11 to 14. In the second and third embodiments, the shadow image described above is used to correct shadow portions in an RGB image and a distance image. In the second embodiment, shadow correction is performed on the RGB image, and in the third embodiment, shadow correction is performed on the distance image. Below, matters common to the second and third embodiments will be described.
[0089] 11 is a functional configuration diagram showing an example of the functional configuration of a shadow area detection system according to the second embodiment. An example of the functional configuration of a shadow area detection system 1A will be described with reference to the same figure. The shadow area detection system 1A differs from the shadow area detection system 1 in that it further includes a shadow correction unit 161. In the description of the shadow area detection system 1A, components similar to those in the shadow area detection system 1 are denoted by similar reference numerals, and descriptions thereof may be omitted.
[0090] The shadow correction unit 161 acquires a SHADOW signal from the shadow candidate 2 detection unit 152, acquires a resized distance image from the resizing unit 113, and acquires an RGB signal from the image acquisition unit 111. The shadow correction unit 161 corrects shadow areas by replacing at least one of the pixel values or distance values of coordinates estimated as shadow candidates by the shadow candidate detection unit 150 with at least one of the pixel values or distance values of coordinates not estimated as shadow candidates. Specifically, based on the acquired information, the shadow correction unit 161 calculates, in both the horizontal and vertical directions, how many pixels each pixel is separated from a non-shadow area having the same OBJ_INDEX number. Specifically, if there is a pixel in a shadow area with a SHADOW signal of 1 in either the left or right horizontal direction that has the same OBJ_INDEX number, the shadow correction unit 161 calculates how many pixels the pixel is separated from the pixel in the horizontal direction, and sets the calculated value as SHADOW_HDISTANCE. At this time, to identify the left and right direction, a sign is added so that if it is on the right, it is positive, and if it is on the left, it is negative. If there are objects with the same OBJ_INDEX number in the left and right horizontal direction, the number with the closest distance is entered into SHADOW_HDISTANCE. Also, if there are no objects with the same OBJ_INDEX number in the left and right direction, SHADOW_HDISTANCE is set to 0.
[0091] Furthermore, if there is a pixel in the shadow area with a SHADOW signal of 1 that has the same OBJ_INDEX number either above or below the pixel in the vertical direction, the shadow correction unit 161 calculates how many pixels it is vertically away from, and sets the calculated value as SHADOW_VDISTANCE. At this time, to indicate the up / down direction, a sign is assigned so that if it is below, it is positive, and if it is above, it is negative. If there are pixels with the same OBJ_INDEX number above or below the pixel in the vertical direction, the number with the closest distance is entered into SHADOW_VDISTANCE. Furthermore, if there is no pixel with the same OBJ_INDEX number above or below, SHADOW_VDISTANCE is set to 0.
[0092] 12A and 12B are diagrams for explaining the shadow correction process performed by the shadow correction unit according to the second embodiment. Fig. 12A shows an example of SHADOW_HDISTANCE calculated according to the above method. Fig. 12B shows the value of SHADOW_VDISTANCE calculated according to the above method.
[0093] 13 is a functional configuration diagram showing an example of the functional configuration of a shadow correction unit according to the second embodiment. An example of the functional configuration of the shadow correction unit 161 will be described with reference to the same figure. The shadow correction unit 161 includes a shadow distance calculation unit 1611, a low-pass filter 1612, a subtraction unit 1613, a substitution unit 1614, a low-pass filter 1615, and an addition unit 1616.
[0094] The shadow distance calculation unit 1611 calculates SHADOW_HDISTANCE and SHADOW_VDISTANCE using the method described above, and outputs the calculated values to the low-pass filter 1612, the substitution unit 1614, and the low-pass filter 1615.
[0095] The replacement unit 1614 replaces the non-shadow portion of the distance pixel count in the direction where the absolute values of SHADOW_HDISTANCE and SHADOW_VDISTANCE are closer with an RGB signal or distance signal. Here, the direction is determined by the signs of SHADOW_HDISTANCE and SHADOW_VDISTANCE. However, if either SHADOW_HDISTANCE or SHADOW_VDISTANCE contains a 0, the non-shadow portion of the distance is replaced with the non-zero direction and the RGB signal or distance signal.
[0096] The low-pass filter 1615 filters the output of the substitution unit 1614. The low-pass filter 1615 changes the filtering level depending on the values of SHADOW_HDISTANCE and SHADOW_VDISTANCE. The longer the distance, the more the low-pass filter 1615 filters, making the image smoother.
[0097] 14 is a functional configuration diagram showing an example of the functional configuration of an LPF according to the second embodiment. An example of the functional configuration of the low-pass filter LPF will be described with reference to the same figure. The low-pass filter 1612 and the low-pass filter 1615 both have the functional configuration shown in the figure, and may have the same functional configuration. Hereinafter, when there is no need to distinguish between the low-pass filter 1612 and the low-pass filter 1615, they may be referred to as the low-pass filter LPF.
[0098] The low-pass filter LPF includes 3TAP horizontal LPFs 411 to 414, an HSEL 42, 3TAP vertical LPFs 431 to 434, and a VSEL 44. The low-pass filter LPF first performs horizontal LPF processing on the RGB signal or the distance signal. The HSEL 42 switches between a through state (not passing through the 3TAP horizontal LPF), a state (passing once, twice, three times, or four times) based on the absolute value of SHADOW_HDISTANCE. Specifically, the HSEL 42 may switch between a through state when SHADOW_HDISTANCE is 0, a state (passing once) when SHADOW_HDISTANCE is 1, a state (passing twice) when SHADOW_HDISTANCE is 2, and a state (passing three times) when SHADOW_HDISTANCE is 3. If SHADOW_HDISTANCE is 4 or more, HSEL 42 may be switched to a state where four passes have been made.
[0099] Each of the 3TAP horizontal LPFs 411 to 414 may be as shown in the following equation (9).
[0100] H3TAPLPF=0.25Z^(-1)+0.5Z^0+0.25Z^1 (9) where Z is the scan time for one pixel.
[0101] The same processing is performed for the vertical direction as for the horizontal direction. Specifically, the VSEL 44 switches the state to one of a through state (not passing through the 3TAP vertical LPF), a state (passing once, passing twice, passing three times, or passing four times) based on the absolute value of SHADOW_VDISTANCE. Specifically, the VSEL 44 may switch the state as follows: through when SHADOW_VDISTANCE is 0, passed once when SHADOW_VDISTANCE is 1, passed twice when SHADOW_VDISTANCE is 2, and passed three times when SHADOW_VDISTANCE is 3. Note that the VSEL 44 may switch to the four-pass state when SHADOW_VDISTANCE is 4 or greater.
[0102] Each of the 3TAP vertical LPF 431 to the 3TAP horizontal LPF 434 may be as shown in the following equation (10).
[0103] V3TAPLPF=0.25Z^(-1)+0.5Z^0+0.25Z^1 (10) where Z is the scan time for one line.
[0104] The low-pass filter 1615 outputs the signal thus subjected to the LPF as SHADOW_LPF.
[0105] The shadow correction unit 161 may include a low-pass filter 1612, which may apply a similar LPF to the RGB signal or distance signal in an unsubstituted state (before passing through the replacement unit 1614). The signal that has passed through the low-pass filter 1612 is subtracted by a subtraction unit 1613 from the RGB signal or distance signal before passing through the low-pass filter 1612, thereby extracting a high-frequency signal. The subtracted signal is added by an addition unit 1616 to the SHADOW_LPF output from the low-pass filter 1615, and the result is output. By adding the high-frequency components extracted from the original signal in this way, it is possible to prevent a loss of detail even when shadow correction is performed.
[0106] The shadow correction unit 161 performs shadow correction using the above-described process, and stores the RGB signal or distance signal with the influence of the shadow reduced in the storage device 22. The correction process performed by the shadow correction unit 161 may be a correction process for either the RGB value or the distance value. The distance signal obtained from the distance measurement sensor 31 tends to show inaccurate distance accuracy in dark areas such as shadows, so correcting only the distance value (without correcting the RGB image) is effective for the purpose of improving distance accuracy in shadow areas.
[0107] The RGB image, distance image, and shadow image according to this embodiment may be captured as continuous frames and treated as a moving image.
[0108] Summary of Embodiments 2 and 3 According to the above-described embodiments, the shadow area detection device 10A uses the shadow candidate detection unit 150 to replace pixel values at coordinates estimated as shadow candidates with values based on pixel values at coordinates that are not estimated as shadow candidates. Therefore, according to this embodiment, it is possible to detect areas of the subject where color information is darkened due to the influence of a shadow and correct the areas based on the RGB values of non-shadow areas. Furthermore, according to this embodiment, it is possible to correct the distance value of a shadow area based on the distance value of a non-shadow area.
[0109] [Fourth Embodiment] Next, a fourth embodiment will be described with reference to FIGS. 15 and 16 . In this embodiment, an imaging device including the shadow area detection system according to any one of the first to third embodiments will be described. Here, when the RGB sensor, polarization sensor, and TOF sensor are photographed using the same lens and on the same optical axis, there is no misalignment between the pixels. By adopting such a configuration, more accurate shadow detection and shadow correction are possible. In this embodiment, an example will be described in which the RGB sensor, polarization sensor, and TOF sensor are arranged on the same optical axis using a prism.
[0110] 15 is a cross-sectional view showing a first example of an arrangement of an image capture device including a shadow area detection system according to the third embodiment. An image capture device 51, which is the first example of the arrangement, will be described with reference to the same figure. The image capture device 51 includes a depolarizer 511, a polarization sensor 512, a non-polarizing half mirror 513, a visible light / infrared separation dichroic coat 514, a ToF sensor 515, and an RGB sensor 516.
[0111] As shown in the figure, the polarization sensor 512 is disposed on the first exit surface, which is not affected by polarization changes due to the visible light / infrared separation dichroic coat 514. Furthermore, by disposing a depolarizing plate 511 on the first exit surface, changes in polarization state due to total reflection within the first prism are eliminated. A non-polarizing half mirror 513 is also used as the half mirror. Specifically, the non-polarizing half mirror 513 may be formed of a metal film or a dielectric film. The visible light / infrared separation dichroic coat 514 separates visible light and near-infrared light. As a result of the separation, a ToF sensor 515 is disposed on the third exit surface through which near-infrared light passes, and an RGB sensor 516 is disposed on the second exit surface through which visible light passes.
[0112] 16 is a cross-sectional view showing a second example of an arrangement of an image capture device including a shadow area detection system according to the third embodiment. An image capture device 52, which is the second example of the arrangement, will be described with reference to the same figure. The image capture device 52 includes a non-polarized visible light / infrared light separation dichroic coat 521, a ToF sensor 522, a non-polarized half mirror 523, a polarization sensor 524, and an RGB sensor 525.
[0113] As shown in the figure, in the imaging device 52, the polarization sensor 524 is disposed on the third exit surface that is not affected by total reflection of the first prism. A non-polarizing half mirror 523 is used as the half mirror. Specifically, the non-polarizing half mirror 523 may be formed of a metal film or a dielectric film. A non-polarized dichroic coat that suppresses phase differences due to wavelengths is used as the non-polarized visible light / infrared separation dichroic coat 521. A ToF sensor 522 is disposed on the first exit surface through which near-infrared light reflected by the non-polarized visible light / infrared separation dichroic coat 521 passes, and an RGB sensor 525 is disposed on the second exit surface through which visible light passes.
[0114] [Summary of the Fourth Embodiment] According to the embodiment described above, by using the prism configuration as described above, when configuring an imaging device equipped with the shadow region detection system according to any one of the above-described Embodiments 1 to 3, the RGB sensor, polarization sensor, and ToF sensor can each capture images using the same lens and the same optical axis. This makes it possible to reduce the misalignment of each pixel, enabling more accurate shadow detection and shadow correction.
[0115] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
[0116] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.
[0117] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate.
[0118] According to the present invention, it is possible to detect portions of a subject where the color information is dark due to the influence of a shadow.
[0119] 1...shadow area detection system, 10...shadow area detection device, 111...image acquisition unit, 113...resizing unit, 115...hue / brightness calculation unit, 116...distance area division unit, 117...color area division unit, 118...object area division unit, 119...area division unit, 131...polarized image acquisition unit, 132...polarization angle calculation unit, 133...resizing unit, 150...shadow candidate detection unit, 151...shadow candidate 1 detection unit, 152...shadow candidate 2 detection unit, 21...storage device, 22...storage device, 31...range measurement sensor, 32...RGB sensor, 33...polarization sensor, 161...shadow correction unit, 1611...shadow distance calculation unit, 1612...low-pass filter, 1613...subtraction unit, 1614...replacement unit, 1615...low-pass filter, 1616...addition unit, 51...image capture device, 52...image capture device
Claims
1. A shadow area detection device comprising: an image acquisition unit that acquires an image having information about a subject at each coordinate on a two-dimensional plane; a region division unit that divides each coordinate of the acquired image into a plurality of regions based on the information about the subject that each coordinate has; a polarized image acquisition unit that acquires a plurality of polarized images of the subject, each of which is captured through a plurality of polarizing filters having a plurality of different polarization angles; and a shadow candidate detection unit that detects candidates for coordinates estimated to be shadow areas for each coordinate of the image, based on information about the image regions divided by the region division unit and the polarized images.
2. The shadow area detection device according to claim 1, wherein the image acquired by the image acquisition unit includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane, and the area division unit further comprises a color area division unit which divides each coordinate of the RGB image into a plurality of areas based on the color information possessed by each coordinate.
3. The shadow region detection device according to claim 2, further comprising a shadow correction unit that replaces pixel values of coordinates estimated to be shadow candidates by the shadow candidate detection unit with values based on pixel values of coordinates that are not estimated to be shadow candidates.
4. A shadow area detection device as claimed in any one of claims 1 to 3, wherein the image acquired by the image acquisition unit includes a distance image having information relating to the distance to the subject at each coordinate on a two-dimensional plane, and the area division unit further comprises a distance area division unit which divides each coordinate of the distance image into a plurality of areas based on the distance information possessed by each coordinate.
5. A shadow area detection method comprising: an image acquisition step of acquiring an image having information of a subject at each coordinate on a two-dimensional plane; a region division step of dividing each coordinate of the acquired image into a plurality of regions based on the information of the subject at each coordinate; a polarized image acquisition step of acquiring a plurality of polarized images of the subject, each of which is captured through a plurality of polarizing filters having different polarization angles; and a shadow candidate detection step of detecting candidates for coordinates estimated to be shadow areas for each coordinate of the image, based on information about the image regions divided by the region division step and the acquired polarized images.
Citation Information
Patent Citations
Image processing apparatus and method, and imaging device
JP2017123069A
Image processing apparatus and method, and imaging device
JP2018098583A
Three-dimensional shape measuring method and three-dimensional shape measuring device
JP2022070696A
Image processing device and image processing method
JP2023005997A
Normal information generating device and normal information generating method
WO2009019886A1
