Shadow region detection device and shadow region detection method
The shadow area detection device uses image analysis and polarization filters to identify and correct shadow areas in depth camera data, addressing the challenge of inaccurate color correction in shadow regions, thereby improving three-dimensional model synthesis in metaverse spaces.
Patent Information
- Application Number
- JP2023081280
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-28
AI Technical Summary
Conventional depth cameras struggle to accurately correct color information in shadow areas, leading to discomfort when synthesizing three-dimensional models in metaverse spaces, as it is difficult to distinguish between shadow-induced darkness and original subject color.
A shadow area detection device that utilizes an image acquisition unit, area dividing unit, and polarization filters to detect and correct shadow areas by analyzing RGB and distance images, employing polarization images to determine shadow candidates and correct pixel values.
Accurately identifies and corrects shadow areas, improving the appearance of three-dimensional models by distinguishing between shadow-induced darkness and original subject color, enhancing the accuracy of distance and color information.
Smart Images

Figure 2024165249000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a shadow area detection device and a shadow area detection method. [Background technology]
[0002] Conventionally, there has been a depth camera (RGBD camera) capable of acquiring color information of a subject and distance information to the subject. Based on information acquired using such a depth camera, a three-dimensional model such as three-dimensional point cloud data or mesh data is generated. The generated three-dimensional model is, for example, synthesized in a metaverse space. In the metaverse space, a shadow due to the presence of another three-dimensional model synthesized in the same metaverse space may be added to the three-dimensional model. Here, when a three-dimensional model of a subject existing in the real world is generated using a depth camera, if a shadow due to the ambient light at the time of shooting exists, the direction of the light source existing in the metaverse space does not match, and therefore the appearance may become unnatural. In addition, there was a problem that the distance accuracy of the dark image in the shadow part cannot be obtained according to the distance information obtained using the depth camera. Therefore, in order to improve the accuracy of the distance information measured by the depth camera, there was a technology to generate and correct a normal vector using information from a polarization sensor (see Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2016 / 136086 Summary of the Invention [Problem to be solved by the invention]
[0004] However, according to the above-mentioned conventional technology, although it may be possible to improve the accuracy of the distance information, it is not possible to correct the color information (RGB information). Therefore, there is a problem that the shadow is superimposed on the generated three-dimensional model, and when it is synthesized in the metaverse space, the part that is dark due to the influence of the shadow (hereinafter referred to as the shadow part) creates an unnatural feeling. Here, in order to reduce the unnaturalness in appearance due to the shadow, it is possible to correct the color information of the shadow part that is darker than the surrounding color. However, even if the part is darker than the surrounding color, it is difficult to determine whether the part is darker due to the influence of the shadow or is the original color of the subject, and it has been difficult to correct the color information of the shadow part.
[0005] The present invention has been made in consideration of the above circumstances, and has an object to provide a shadow area detection device capable of detecting areas of a subject where color information is darkened due to the influence of a shadow. [Means for solving the problem]
[0006] [1] One aspect of the present invention is a shadow area detection device that includes an image acquisition unit that acquires an image having information about a subject at each coordinate on a two-dimensional plane; a region division unit that divides each coordinate of the acquired image into a plurality of regions based on information about the subject at each coordinate; a polarized image acquisition unit that acquires a plurality of polarized images of the subject, each imaged through a polarizing filter having a plurality of different polarization angles; and a shadow candidate detection unit that detects candidate coordinates that are estimated to be shadow areas for each coordinate of the image based on information about the image regions divided by the region division unit and the polarized images.
[0007] [2] Moreover, according to one aspect of the present invention, in the shadow region detection device described in [1] above, the image acquired by the image acquisition unit includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane, and the region division unit further includes a color region division unit that divides each coordinate of the RGB image into a plurality of regions based on the color information possessed by each coordinate.
[0008] [3] Furthermore, one aspect of the present invention is the shadow area detection device described in [2] above, further comprising a shadow correction unit that replaces pixel values of coordinates that have been estimated as shadow candidates by the shadow candidate detection unit with values based on pixel values of coordinates that have not been estimated as shadow candidates.
[0009] [4] Furthermore, one aspect of the present invention is a shadow area detection device as described in any one of [1] to [3] above, wherein the image acquired by the image acquisition unit includes a distance image having information regarding a distance to the subject at each coordinate on a two-dimensional plane, and the area division unit further includes a distance area division unit that divides each coordinate of the distance image into a plurality of areas based on the distance information possessed by each coordinate.
[0010] [5] Also, one aspect of the present invention includes an image acquisition step of acquiring an image having information of a subject at each coordinate on a two-dimensional plane; a region dividing step of dividing each coordinate of the acquired image into a plurality of regions based on information of the subject held by each coordinate; a polarization image acquisition step of acquiring a plurality of polarization images of the subject through a plurality of polarization filters each having a different polarization angle; a shadow candidate detection step of detecting candidates of coordinates estimated to be a shadow portion for each coordinate of the image based on information on the image regions divided by the region division step and the acquired polarized image; The shadow region detection method has the following structure. Effect of the Invention
[0011] According to the present invention, it is possible to detect a portion of a subject where color information is darkened due to the influence of a shadow. [Brief description of the drawings]
[0012] [Figure 1] 2A to 2C are diagrams for explaining a shadow detected by the shadow region detection system according to the first embodiment. [Diagram 2] 2 is a functional configuration diagram showing an example of the functional configuration of the shadow region detection system according to the first embodiment. FIG. [Diagram 3] 4A to 4C are diagrams for explaining a distance area division process performed by a distance area division unit according to the first embodiment. [Figure 4A] 4 is a first flowchart showing an example of a distance area division process by a distance area division section according to the first embodiment. [Figure 4B] 11 is a second flowchart showing an example of distance area division processing by the distance area division unit according to the first embodiment. [Diagram 5] 4A to 4C are diagrams for explaining a color region dividing process performed by a color region dividing section according to the first embodiment; [Figure 6A] 4 is a first flowchart showing an example of a color region dividing process performed by a color region dividing section according to the first embodiment. [Figure 6B] 11 is a second flowchart showing an example of a color region dividing process performed by the color region dividing section according to the first embodiment. [Figure 7] 4A to 4C are diagrams for explaining an object region division process performed by an object region division section according to the first embodiment. [Figure 8] 1A and 1B are diagrams for explaining an example of the configuration of a polarization sensor according to a first embodiment. [Figure 9] 13 is an example of a case where images are color-coded based on a signal of a polarization angle obtained by the polarization sensor according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a SHADOW image according to the first embodiment. [Figure 11] FIG. 11 is a functional configuration diagram showing an example of the functional configuration of a shadow region detection system according to a second embodiment. [Figure 12] 13A to 13C are diagrams for explaining a shadow correction process performed by a shadow correction unit according to the second embodiment. [Figure 13] FIG. 11 is a functional configuration diagram showing an example of the functional configuration of a shadow correction unit according to the second embodiment. [Figure 14] FIG. 11 is a functional configuration diagram showing an example of the functional configuration of an LPF according to a second embodiment. [Figure 15] 11 is a cross-sectional view showing a first example of arrangement of an imaging device including a shadow region detection system according to a third embodiment. FIG. [Figure 16] 13 is a cross-sectional view showing a second example of arrangement of an imaging device including a shadow region detection system according to a third embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] A shadow region detection device according to an aspect of the present invention will be described in detail below with reference to the accompanying drawings, showing a preferred embodiment. Note that the embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. In addition, "based on XX" in this application means "based on at least XX", and includes cases based on other elements in addition to XX. In addition, "based on XX" is not limited to cases where XX is directly used, but also includes cases where XX is based on a calculation or processing performed on XX. "XX" is any element (for example, any information). In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each configuration easier to understand.
[0014] [First embodiment] First, a shadow region detection system 1 according to a first embodiment will be described with reference to FIGS.
[0015] FIG. 1 is a diagram for explaining a shadow detected by a shadow region detection system according to the first embodiment. FIG. 1(A) shows an example of a case where an RGB image is captured of an object A as a subject. The figure shows the angle of view of the captured RGB image. Here, when an RGB image is captured with the angle of view shown in the figure, it is assumed that an object B exists outside the angle of view. In this case, it can be seen that a shadow is projected in the RGB image of object A due to the influence of a light source such as the sun and object B. The shadow portion is darker (i.e., has lower brightness) than the original color information of object A.
[0016] FIG. 1(B) shows object B, which has a similar shape to object A, but different color information. The figure shows the color information that object B originally has. In other words, object B has color information as shown, without the influence of shadows. The color information of object A shown in FIG. 1(A) and the color information of object B shown in FIG. 1(B) are identical or similar to each other. In this way, just by looking at an RGB image, it may be difficult to determine whether a part that is darker than the surroundings (i.e., a part with low brightness) is the original color of the subject, or the color of a shadow cast by an object not shown in the image, as shown in FIG. 1(A).
[0017] When generating a three-dimensional model that is intended to be synthesized in a virtual space such as the metaverse in the future, if it is known which parts of the color information of the object being photographed are shadow parts, it is possible to generate a three-dimensional model after correcting the color information. The shadow region detection system 1 according to this embodiment can detect shadow regions from the color information obtained by capturing an image of an object.
[0018] 2 is a functional configuration diagram showing an example of the functional configuration of the shadow area detection system according to the first embodiment. With reference to the diagram, an example of the functional configuration of the shadow area detection system 1 will be described. The shadow area detection system 1 includes a shadow area detection device 10, a storage device 21, a storage device 22, a distance measurement sensor 31, an RGB sensor 32, and a polarization sensor 33.
[0019] The storage device 21 stores parameters and the like required for processing by the shadow area detection device 10. The parameters and the like stored in the storage device 21 may be rewritten as appropriate by a user or the like, and may change according to environmental information such as the time of day, temperature, or weather. The storage device 22 stores information for identifying the shadow area detected by the shadow area detection device 10. Both the storage device 21 and the storage device 22 may be included in the shadow area detection device 10.
[0020] The distance measurement sensor 31 acquires three-dimensional information of the subject. Specifically, the distance measurement sensor 31 acquires a distance image by measuring the distance to the subject at multiple points. The distance measurement sensor 31 may be, for example, a sensor that uses a ToF (Time of Flight) method to two-dimensionally irradiate the subject with light and measure the distance based on the time it takes to receive the reflected light. The distance measurement sensor 31 outputs the acquired distance image to the shadow area detection device 10.
[0021] The RGB sensor 32 captures an RGB image of the same subject as the subject to be measured by the distance measurement sensor 31. It is preferable that the coordinates of the distance image acquired by the distance measurement sensor 31 and the coordinates of the RGB image acquired by the RGB sensor 32 correspond to each other. The distance measurement sensor 31 and the RGB sensor 32 may be components of a depth camera (RGBD camera) not shown. The RGB sensor 32 includes a plurality of photodiodes having filters of each RGB color arranged in a Bayer pattern, for example, and acquires luminance information of each RGB color. The RGB sensor 32 outputs the acquired RGB image to the shadow area detection device 10 as, for example, a RAW image.
[0022] The polarization sensor 33 is the target of distance measurement by the distance measurement sensor 31 and acquires a polarized image of the same subject as the subject to be imaged by the RGB sensor 32. The polarization sensor 33 captures a plurality of polarized images captured through a plurality of polarizing filters (polarizers) having different polarization angles. The polarization sensor 33 outputs the acquired polarized images to the shadow area detection device 10.
[0023] The shadow region detection device 10 includes an image acquisition unit 111, a resizing unit 113, a hue / luminance calculation unit 115, a region division unit 119, a polarized image acquisition unit 131, a polarization angle calculation unit 132, a resizing unit 133, and a shadow candidate detection unit 150. Each of these functional units is realized, for example, by using an electronic circuit. Furthermore, each functional unit may include internal storage means such as a semiconductor memory or a magnetic hard disk device as necessary. Furthermore, each function may be realized by a computer having a CPU (Central Processing Unit) and software.
[0024] The image acquisition unit 111 acquires an image having information about a subject at each coordinate on a two-dimensional plane. The image includes an RGB image having information about the color of the subject at each coordinate on the two-dimensional plane, and a distance image having information about the distance to the subject at each coordinate on the two-dimensional plane. That is, the image acquisition unit 111 acquires at least one of an RGB image and a distance image. Specifically, the image acquisition unit 111 acquires an RGB image from the RGB sensor 32. The image acquisition unit 111 performs a de-Bayer (demosaic) process on the acquired RGB image to obtain image signals of each color of RGB. The image acquisition unit 111 also acquires a distance image from the distance measurement sensor 31.
[0025] Resizing unit 113 acquires information about the distance image from image acquisition unit 111. When the resolution of the distance image differs from the resolution of each RGB color acquired from the RGB image, resizing unit 113 resizes (upconverts) the distance image to match the resolution of each. Note that resizing unit 113 may acquire image signals of each RGB color from image acquisition unit 111 and resize (downconvert) the image signals of each RGB color to match the resolution of each.
[0026] Region dividing section 119 acquires a distance image and an RGB image, both of which have been adjusted to the same resolution by resizing section 113. Region dividing section 119 divides each coordinate of the acquired image into a plurality of regions based on information of the subject held by each coordinate. Specifically, region dividing section 119 includes distance region dividing section 116, color region dividing section 117, and object region dividing section 118. Distance region dividing section 116 performs region dividing processing based on the distance image. It can also be said that distance region dividing section 116 divides each coordinate of the distance image into a plurality of regions based on distance information held by each coordinate. Color region dividing section 117 performs region dividing processing based on the RGB image. It can also be said that color region dividing section 117 divides each coordinate of the RGB image into a plurality of regions based on color information held by each coordinate. The object region dividing section 118 performs a region dividing process based on the region dividing result performed by the distance region dividing section 116 and the region dividing result performed by the color region dividing section 117. The region dividing process is a process of dividing each coordinate of an image into regions for each of a plurality of objects present in the image. In the region dividing process, the same identification number (INDEX number) is assigned to regions of similar colors within the same object.
[0027] First, a description will be given of the region division process performed by distance region division unit 116. Distance region division unit 116 acquires a distance image resized by resizing unit 113 and adjusted to the same resolution as the image signals of each color of RGB. Distance region division unit 116 also acquires from storage device 21 parameters DTH1max, DTH1min, DTH2, and DTH3 used in the distance region division process.
[0028] DTH1max is a distance for removing distance data of the background of the subject from the shadow detection area. In other words, DTH1max can be said to be the maximum value in the depth direction of the subject. DTH1min is a distance for removing distance data in front of the subject from the shadow detection area. In other words, DTH1min can be said to be the maximum value in the front direction of the subject. DTH2 is a threshold for determining whether or not the distance values of adjacent coordinates are on the same plane. DTH3 is a threshold for removing abnormal values based on large random noise caused by the sensor and dead pixels, etc. The distance area division unit 116 uses these parameters to perform a division process on the object area of the subject and assigns a distance INDEX number to each area. This is based on the idea that if the object is the same, the distance values of adjacent pixels are not far apart. The distance INDEX number is a number assigned to each pixel based on distance information, and the same number is assigned to pixels that are estimated to be the same object.
[0029] The operation of assigning distance INDEX numbers by the distance area division unit 116 will be described in detail with reference to FIGS. 3, 4A, and 4B.
[0030] FIG. 3 is a diagram for explaining the distance area division process by the distance area division unit according to the first embodiment. In the figure, the distance image (depth data image) acquired by the image acquisition unit 111 is shown with a small number of pixels for the purpose of explanation. In the figure, the value of the distance image (distance value) is shown by the shade of gray. Specifically, the darker the color, the closer the object is, and the lighter the color, the farther the object is. FIG. 3(A) shows each coordinate with a number added to identify each coordinate for explanation. FIG. 3(B) shows the distance INDEX number added by the distance area division unit 116.
[0031] Figures 4A and 4B (hereinafter sometimes simply referred to as Figure 4) are a flowchart showing an example of distance area division processing by the distance area division unit according to the first embodiment. The illustrated flowchart shows the flow of processing when sequential processing is performed by scanning in the horizontal and vertical directions from the upper left corner pixel of the distance image shown in Figure 3. The flowchart shown in Figure 4 loops so as to process by sequential scanning as indicated by the arrows shown in Figure 3.
[0032] First, the distance area division unit 116 acquires the distance value (depth data) of a 3×3 pixel area from the distance screen (step S111). Here, the acquired 3×3 pixels may include data that is different from the surroundings by only one pixel due to sensor noise or dead pixels. The distance area division unit 116 first assigns a distance INDEX number to such data in steps S112 and S113. Specifically, the distance area division unit 116 compares only the center of the 3×3 pixels with all the surrounding pixels and determines whether the difference in distance value is +DTH3 or more or -DTH3 or less (step S112). If the distance area division unit 116 determines that there is a predetermined difference in the distance value (i.e., step S112; YES), it sets the distance INDEX number of the center pixel of the 3×3 pixels to the distance INDEX number of the neighboring pixel on the left (step S113). Note that if a distance INDEX number has not yet been assigned to the top left 3×3 area, the distance INDEX number is set to 0. When the distance area dividing unit 116 determines that there is no predetermined difference in the distance values (ie, step S112; NO), the process proceeds to step S114.
[0033] Looking at FIG. 3(A), there is a coordinate labeled F in the center. Assume that the difference in distance value between the pixel at that coordinate and the surrounding area is ±DTH3. In such a case, the pixel at that coordinate becomes the distance INDEX number of the pixel to the left. Specifically, as shown in FIG. 3(B), the distance INDEX number is determined to be 1, just like the distance INDEX number of the pixel to the left. The determined distance INDEX number is stored in a temporary storage unit (not shown), and the process moves to the next pixel and loops to the beginning of the flow. Note that in the process described below, a distance INDEX number is assigned to each pixel, and the assigned distance INDEX numbers are stored in the temporary storage unit.
[0034] If step S112 is NO, then in steps S114 and S115, the distance area division unit 116 performs a process of excluding backgrounds present behind the subject and unnecessary objects present in front of the subject from targets for shadow detection. Specifically, the distance area division unit 116 performs a determination using distance thresholds DTH1max and DTH1min. DTH1max is a threshold for excluding objects farther from the subject from targets for shadow detection. Also, DTH1min is a threshold for excluding objects closer to the subject from targets for shadow detection. First, the distance area division unit 116 determines whether the distance value of the center pixel of 3×3 pixels is equal to or smaller than DTH1min or equal to or larger than DTH1max (step S114). If the distance value exceeds the threshold (i.e., step S114; YES), the distance area division unit 116 sets the distance INDEX number of the center pixel to 0 (step S115). Furthermore, if the distance value does not exceed the threshold (i.e., step S114; NO), distance area division unit 116 advances the process to step S116. Specifically, in the example shown in FIG. 3(A), coordinates without numbers indicate parts that exceed DTH1max, that is, parts that are determined to be background behind the subject. Also, parts numbered 33 to 39 indicate parts that are equal to or less than DTH1min, that is, parts that are determined to be extraneous objects in front of the subject. Such parts have a distance INDEX number of 0, as shown in FIG. 3(B).
[0035] If step S114 is NO, then in steps S116 and S117, the distance area dividing unit 116 assigns a distance INDEX number to the newly detected object. In other words, the distance area dividing unit 116 can also assign a new distance INDEX number to the contour of the object. Specifically, since the distance area dividing unit 116 sequentially scans from the upper left pixel in the horizontal direction, it can be said that when scanning a certain pixel, the upper three pixels and the left one pixel out of the 3×3 pixels have already been scanned. The distance area dividing unit 116 determines whether or not there is still a distance INDEX number other than 0 among the pixels that have been scanned in this way (step S116). If there is still no distance INDEX number other than 0 (i.e., step S116; YES), the distance area dividing unit 116 assigns a distance INDEX number to the pixel (step S117). Furthermore, if there is still a distance INDEX number other than 0 (i.e., step S116; NO), distance area division unit 116 advances the process to step S118. By performing such an operation, distance area division unit 116 can assign distance INDEX numbers to the central pixels of the 3×3 pixel array to which no distance INDEX number has been assigned so far. In the illustrated example, the pixel assigned with 1 in FIG. 3(A) is the first pixel that has been determined to have an object after scanning from the top left, and so is assigned a distance INDEX number of 1. Moreover, the pixel assigned with 22 is assigned in a situation where distance INDEX numbers up to 3 have already been assigned, so is assigned a distance INDEX number of 4.
[0036] If step S116 is NO, then the distance area division unit 116 determines whether the parts determined to be objects exist on the same plane in steps S118 to S121. That is, the distance area division unit 116 determines whether to assign the same distance INDEX number or different distance INDEX numbers in steps S118 to S121. Here, the processing of step S118 is performed when the top three pixels and the left one pixel of the 3×3 pixels that have already been scanned and assigned distance INDEX numbers have distance INDEX numbers other than 0. In such a case, the distance area division unit 116 determines whether the center pixel of the 3×3 pixels is on the same plane as the pixel to which the distance INDEX number has already been assigned, based on the threshold level of DTH2.
[0037] Specifically, the distance area division unit 116 judges whether any of the distance values of the top three pixels and the left one pixel is within −DTH2 to +DTH2 of the distance value of the central pixel of the 3×3 (step S118). If any of the distance values of the top three pixels and the left one pixel is within −DTH2 to +DTH2 of the distance value of the central pixel of the 3×3 (i.e., step S118; YES), the distance area division unit 116 regards them as the same plane and assigns the same distance INDEX number to each of them. At this time, if there are multiple pixels whose distance value is within −DTH2 to +DTH2 of the central pixel, all of those pixels are assigned the smallest distance INDEX number among them (step S119). Also, if the distance INDEX numbers have already been assigned to the top three pixels and the left one pixel and the distance INDEX numbers are rewritten, the coordinates that have been scanned and have distance INDEX numbers stored are referred to, and all of the coordinates having the same distance INDEX number are rewritten (step S121).
[0038] In the illustrated example, the pixels with coordinates 2, 6, 12, and 13 in FIG. 3A had distance INDEX numbers of 2 until the scanning process was performed on the pixel with coordinate 18, but when the pixel with coordinate 18 was judged, the distance INDEX number was reassigned to 1. This is because the distance value of the pixel with coordinate 17 was within the range of -DTH2 to +DTH2. At that time, as shown in FIG. 3B, the distance INDEX numbers of the pixels with coordinates 2, 6, 12, and 13 outside the 3×3 pixel area that had already been stored as distance INDEX numbers of 2 are also reassigned to 1. After this processing, the process moves to the next pixel.
[0039] If any of the distance values of the top three pixels and the left pixel is not within −DTH2 to +DTH2 of the distance value of the central pixel of the 3×3 pixel (i.e., step S118; NO), the distance area division unit 116 updates the distance INDEX number of the target pixel to a number other than the distance INDEX number that has been assigned so far, and sets it as a new distance INDEX number (step S120). For example, for a pixel with coordinate 7 shown in FIG. 3(A), the distance INDEX number is 2 during the scanning process of this pixel. However, if the distance value between the left pixel (i.e., the pixel with coordinate 6) and the upper left pixel (i.e., the pixel with coordinate 2) is not within −DTH2 to +DTH2, the distance INDEX number is updated and the distance INDEX number is stored as 3 (step S120). After the above process, the distance area division unit 116 proceeds to the process of the next pixel (step S122).
[0040] The distance area division unit 116 scans all pixels in the above manner. That is, in this embodiment, the distance INDEX number is updated as a result of multiple determinations. Figure 3(B) shows the final results for all pixels.
[0041] Returning to FIG. 2, the distance area division unit 116 outputs information on the finally obtained distance INDEX number to the object area division unit 118.
[0042] Next, the region division process performed by color region division unit 117 will be described. Color region division unit 117 acquires RGB color signals based on the RGB image acquired by image acquisition unit 111, and a distance image resized by resizing unit 113 to have the same resolution as the image signals of the RGB colors. Color region division unit 117 also acquires WTH, DTH1min, DTH1max, HUETH, and SATTH, which are parameters used in the color region division process. DTH1min and DTH1max may be the same parameters as those in the distance region division process performed by distance region division unit 116.
[0043] WTH is a threshold value for removing abnormal values based on prominent noise of the RGB sensor, dead pixels, etc. WTH may be a value that varies according to temperature. HUETH is a threshold value for hue for dividing regions by color. SATTH is a threshold value for color saturation. The color region dividing unit 117 uses these parameters to divide the object regions of the subject and assigns a color INDEX number to each region. This is based on the idea that if the same color is in the same object, even if it is a shadow and a non-shadow part, both will fall into a similar hue. The color INDEX number is a number assigned to each pixel based on hue information, and the same number is assigned to pixels that are estimated to be of similar colors in the same object. The distance INDEX number and the color INDEX number are obtained by dividing the same object into a region based on distance and a region based on color, respectively, using different approaches.
[0044] In addition, when the hue cannot be determined, for example, for colorless black, gray, and white objects, the color saturation is small, approximately at the noise level. Therefore, in this embodiment, objects below a certain color saturation level are regarded as the same black, white, and gray objects.
[0045] The operation of assigning color INDEX numbers by color region division section 117 will be described in detail with reference to FIGS. 5, 6A and 6B.
[0046] FIG. 5 is a diagram for explaining the color region division process by the color region division unit according to the first embodiment. In the figure, the RGB image acquired by the image acquisition unit 111 is represented by a small number of pixels for the purpose of explanation. In the figure, pixel values are represented by shades of gray. In FIG. 5(A), a number is assigned to each coordinate to identify each coordinate used for explanation. FIG. 5(B) shows the color INDEX number assigned by the color region division unit 117.
[0047] 6A and 6B (hereinafter, sometimes simply referred to as FIG. 6) are flowcharts showing an example of color region division processing by the color region division unit according to the first embodiment. The illustrated flowchart shows the flow of processing when sequential processing is performed by scanning in the horizontal and vertical directions from the upper left corner pixel of the RGB image shown in FIG. 5. The flowchart shown in FIG. 6 loops so as to process by sequential scanning as indicated by the arrows shown in FIG. 5.
[0048] First, the color region division unit 117 converts the input RGB color signals into luminance Y, hue HUE, and color saturation SAT. For this conversion, for example, the following formulas (1) to (5) may be used.
[0049] Y=0.2126×R+0.7152×G+0.0722×B...(1) Pb=-0.1146×R-0.3854×G+0.5000×B...(2) Pr=0.5000×R-0.4542×G-0.0458×B...(3) HUE = ATA N (Pr / Pb) (4) SAT = SQRT(PB^2+Pr^2) (5)
[0050] Using equations (1) to (5), color region division unit 117 calculates HUE, SAT, and Y for all 3×3 pixels (step S211). Color region division unit 117 also obtains a distance value (Depth value) of the center pixel of the 3×3 pixels.
[0051] Next, in steps S212 and S213, the color region division unit 117 processes pixels having abnormal values due to noises that are particularly large among noises caused by the RGB sensor, dead pixels, etc., and pixels having pixel defects called white defects and black defects. Specifically, the color region division unit 117 first compares the RGB color signals of the central pixel of the 3×3 pixels with the RGB color signals of the 8 surrounding pixels of the 3×3 pixels (step S212). If there is at least one channel in the 8 surrounding pixels whose pixel value is equal to or greater than +WTH or equal to or less than −WTH of the central pixel (i.e., step S212; YES), the color region division unit 117 regards the central pixel as a dead pixel and sets the color INDEX number of the pixel to the left (step S213). Note that if a color INDEX number has not yet been assigned to the top left 3×3 region, the color INDEX number is set to 0.
[0052] In the illustrated example, in FIG. 5(A), the pixel indicated by W has an RGB value that is farther away from the surrounding pixels by ±WTH. In this case, the color region division unit 117 sets the color INDEX number of the pixel indicated by W to the color INDEX number of the adjacent pixel. Looking at FIG. 5(B), the color INDEX number of the pixel indicated by W in FIG. 5(A) is the same as the color INDEX number of the adjacent pixel (i.e., the pixel with coordinate 16). The determined color INDEX number is stored in a temporary storage unit (not shown), and the process moves to the next pixel and loops to the beginning of the flow. Note that in the process described below, a color INDEX number is assigned to each pixel, and the assigned color INDEX number is stored in the temporary storage unit.
[0053] If step S212 is NO, then in steps S214 and S215, color region division unit 117 performs a process of excluding the background behind the subject and unnecessary objects in front of the subject from the targets of shadow detection. Specifically, color region division unit 117 performs a determination using distance thresholds DTH1max and DTH1min. First, color region division unit 117 determines whether the distance value of the center pixel is equal to or less than DTH1min or equal to or more than DTH1max (step S214). If the distance value exceeds the threshold (i.e., step S214; YES), color region division unit 117 sets the color INDEX number of the center pixel to 0 (step S215). If the distance value does not exceed the threshold (i.e., step S214; NO), color region division unit 117 advances the process to step S216. Specifically, in the example shown in Fig. 5(A), coordinates without numbers indicate areas that exceed DTH1max, i.e. areas that are determined to be background areas behind the subject. Areas numbered 35 to 41 indicate areas that are below DTH1min, i.e. areas that are determined to be extraneous objects in front of the subject. Such areas are assigned a color INDEX number of 0, as shown in Fig. 5(B).
[0054] If step S214 is NO, then in step S216 and step S217, color region division unit 117 assigns a color INDEX number to the newly detected object. In other words, it is also possible to assign a new color INDEX number to the contour of the object. Specifically, since color region division unit 117 sequentially scans from the upper left pixel in the horizontal direction, it can be said that when scanning a certain pixel, the upper three pixels and the left one pixel out of the 3×3 pixels have already been scanned. Color region division unit 117 determines whether there is still a color INDEX number other than 0 among the pixels that have been scanned in this way (step S216). If there is still no color INDEX number other than 0 (i.e., step S216; YES), color region division unit 117 assigns a color INDEX number to the pixel (step S217). Furthermore, if there is still a distance INDEX number other than 0 (i.e., step S216; NO), color region division unit 117 advances the process to step S218. By performing such an operation, color region division unit 117 can assign color INDEX numbers to the central pixels of the 3×3 pixel array to which no color INDEX number has been assigned so far. In the illustrated example, the pixel assigned with 1 in FIG. 5(A) is the first pixel that has been determined to have an object when scanning from the top left, and therefore is assigned a distance INDEX number of 1. Moreover, the pixel assigned with 24 is assigned in a situation where distance INDEX numbers up to 3 have already been assigned, and therefore is assigned a distance INDEX number of 4.
[0055] If step S216 is NO, then in steps S218 and S219, the color region division unit 117 judges whether the pixel value is in the vicinity of black and white gray. A threshold value of SATTH is used to judge whether the pixel value is in the vicinity of black and white gray. Specifically, the color region division unit 117 judges whether there is at least one pixel with a color INDEX number of 1 or more in either the top three pixels or the left pixel, and whether the SAT (see the above-mentioned formula (5)) of the top three pixels and the left pixel is within SATTH (step S218). If there is even one pixel that meets this condition (i.e., step S218; YES), these pixels are considered to be of the same color and are assigned the same color INDEX number. At this time, if there are multiple pixels within SATTH, all of those pixels are assigned the smallest distance INDEX number among them (step S219). In the illustrated example, the pixel with 7 and the pixel with 8 in FIG. 5(A) had a color INDEX number of 4 until the scanning process was performed. However, when scanning is performed on the pixel marked with 8, the SAT value of the pixel to the left marked with 7 is within SATTH, so the color INDEX number is reassigned to 2. If a pixel outside the 3×3 region has already been stored with a color INDEX number of 4, the color INDEX number is simultaneously reassigned to 2 and restored. Color region division unit 117 performs this process before moving on to the next pixel.
[0056] If step S218 is NO, then color region division unit 117 determines whether the pixels are of the same hue in steps S220 to S223. That is, color region division unit 117 determines whether to assign the same color INDEX number or different color INDEX numbers in steps S220 to S223. Here, the processing of step S220 is performed when the top three pixels and the left one pixel of the 3×3 pixels that have already been scanned and assigned color INDEX numbers have color INDEX numbers other than 0. In such a case, color region division unit 117 determines whether the central pixel of the 3×3 pixels is of the same hue as the pixel that has already been assigned a color INDEX number, based on the threshold level of HUETH.
[0057] Specifically, color region division unit 117 judges whether any of the HUE values of the top three pixels and the left pixel are within −HUETH to +HUETH of the HUE value of the central pixel of the 3×3 array (step S220). If any of the HUE values of the top three pixels and the left pixel are within −HUETH to +HUETH of the HUE value of the central pixel of the 3×3 array (i.e., step S220; YES), color region division unit 117 regards them as having the same hue and assigns the same color INDEX number to each of them. At this time, if there are multiple pixels whose hue is within −HUETH to +HUETH for the hue of the central pixel, all of these pixels are assigned the smallest color INDEX number among them (step S221). Also, if the color INDEX numbers have already been assigned to the top three pixels and the left pixel and the color INDEX numbers are rewritten, the coordinates that have been scanned and color INDEX numbers stored are referred to, and all of the coordinates having the same color INDEX number are rewritten (step S223).
[0058] In the illustrated example, the pixels with coordinates 6 and 14 in FIG. 5(A) had color INDEX numbers of 3 until the scanning process was performed on the pixel with coordinate 20, but when the pixel with coordinate 20 was judged, the color INDEX number was reassigned to 1. This is because the HUE value of the pixel with coordinate 19 was within the range of -HUETH to +HUETH. At that time, as shown in FIG. 5(B), the color INDEX numbers of the pixels with coordinates 6 and 14 outside the 3×3 pixel area that had already been stored as color INDEX numbers 3 are also reassigned to 1. After this processing, the process moves to the next pixel (step S224).
[0059] If any of the HUE values of the top three pixels and the left pixel is not within -HUETH to +HUETH of the HUE value of the central pixel of the 3x3 (i.e., step S220; NO), the color region dividing unit 117 updates the color INDEX number of the target pixel to a number other than the color INDEX number that has been assigned so far, and sets it as a new color INDEX number (step S222). For example, for a pixel with coordinate 34 in FIG. 5(A), the color INDEX number is 1 during the scanning process of this pixel. However, if the HUE values of the left pixel (i.e., the pixel with coordinate 33), the upper left pixel (i.e., the pixel with coordinate 28), and the upper pixel (i.e., the pixel with coordinate 29) are not within -HUETH to +HUETH, the color INDEX number is updated and the color INDEX number is stored as 5 (step S222). After the process, the distance region dividing unit 116 proceeds to the process of the next pixel (step S224).
[0060] The color region division unit 117 scans all pixels in the above manner. That is, in this embodiment, the color INDEX number is updated as a result of multiple determinations. Figure 5(B) shows the final results for all pixels.
[0061] Returning to FIG. 2, the color region dividing unit 117 outputs information on the finally obtained color INDEX number to the object region dividing unit 118.
[0062] The object region dividing section 118 obtains information about the distance INDEX number from the distance region dividing section 116, and obtains information about the color INDEX number from the color region dividing section 117. The object region dividing section 118 performs final object region division based on the obtained information.
[0063] FIG. 7 is a diagram for explaining the object region division process by the object region division section according to the first embodiment. With reference to the figure, an example of the object region division process performed by the object region division section 118 will be described. The object region division section 118 considers the color INDEX number as a reference, and assigns an OBJ_INDEX number to pixel parts having a color INDEX number other than 0. Here, the reason why the object region division section 118 uses the color INDEX number (rather than the distance INDEX number) as a reference is that the RGB sensor 32 is less susceptible to noise (compared to the distance measurement sensor 31) and the boundaries of objects are relatively smooth. Note that this embodiment is not limited to this example, and the object region division section 118 may consider the distance INDEX number as a reference.
[0064] Object region division unit 118 first assigns OBJ_INDEX numbers to pixels whose color INDEX number is other than 0, with the color INDEX number being the higher order and the distance INDEX number being the lower order. For example, if the color INDEX number is 2 and the distance INDEX number is 3, the OBJ_INDEX number will be 2_3. In this manner, OBJ_INDEX numbers are assigned to all pixels. The result of assigning OBJ_INDEX numbers to all pixels is shown in Figure 7(A).
[0065] Next, the object region division unit 118 scans the OBJ_INDEX number for each 3×3 pixel region, and if the upper digit of the OBJ_INDEX number at the center of the 3×3 pixels is other than 0 and the lower digit is 0, the number that has the largest number of pixels with the same upper digit and the same lower digit among the surrounding 3×3 pixels is assigned, and the number is updated as the OBJ_INDEX number. If this condition does not apply to the surrounding pixels, the number is set to 0_0 and the OBJ_INDEX number is updated. Note that a pixel that applies this condition corresponds to the pixel with the OBJ_INDEX number 2_0 in FIG. 7(A). The pixel is updated to 2_3 as shown in FIG. 7(B). Note that if the condition that the upper digit of the OBJ_INDEX number at the center of the 3×3 pixels is other than 0 and the lower digit is 0 is not applied, the OBJ_INDEX number is not updated. In this way, the OBJ_INDEX number after the final object region division is obtained.
[0066] Returning to Fig. 2, object region division unit 118 outputs the OBJ_INDEX number resulting from the final object region division to shadow candidate 1 detection unit 151 and shadow candidate 2 detection unit 152. In the following description, a configuration including shadow candidate 1 detection unit 151 and shadow candidate 2 detection unit 152 may be referred to as shadow candidate detection unit 150. It can also be said that shadow candidate detection unit 150 detects candidates for coordinates estimated to be shadow parts for each coordinate of the image, based on information about the image regions divided by region division unit 119 and the polarized image acquired by polarized image acquisition unit 131.
[0067] The shadow candidate 1 detection unit 151 then detects candidates estimated to be shadows (hereinafter referred to as shadow candidates) from among the pixel group assigned the same OBJ_INDEX number. It obtains the OBJ_INDEX number from the object area division unit 118 and obtains YTH from the storage device 21. Here, YTH is a threshold value for determining whether there is a difference between the RGB luminance signal Y and the darkness of the shadow. YTH may be a value that varies depending on the time when the image to be processed was captured (i.e., the time when the image acquisition unit 111 acquires the image), the weather, etc.
[0068] The shadow candidate 1 detection unit 151 first scans the OBJ_INDEX numbers from the top left and detects how many pixels with the same OBJ_INDEX number exist for each number. Next, if the number of pixels with the same OBJ_INDEX number is 5 or less, the shadow candidate 1 detection unit 151 treats it as if the object is too small and there is no shadow. In this case, the shadow candidate 1 detection unit 151 sets the value of the shadow candidate 1, which is an output signal, to 0. If the number of pixels with the same OBJ_INDEX number is 6 or more, the shadow candidate 1 detection unit 151 calculates the average of the luminance values of the top three out of the six pixels in order of brightness of luminance Y. If there is a pixel with a luminance value YTH lower than the calculated average luminance value, the shadow candidate 1 detection unit 151 considers the pixel to be a shadow part and sets the value of the shadow candidate 1, which is an output signal, to 1. If there is no pixel with a luminance value YTH lower than the calculated average luminance value, the shadow candidate 1 detection unit 151 considers that there is no shadow and sets the value of the shadow candidate 1 to 0. The shadow candidate 1 detection unit 151 outputs the shadow candidate 1 thus obtained to the shadow candidate 2 detection unit 152. Note that the shadow candidate 1 which is an output signal is a value that exists for each OBJ_INDEX number.
[0069] Next, we will explain the functions of the polarization image acquisition section 131, the polarization angle calculation section 132, and the resizing section 133. As a prerequisite for the functions of the polarization angle calculation section 132 and the resizing section 133, we will explain an example of the configuration of the polarization sensor 33.
[0070] FIG. 8 is a diagram for explaining an example of the configuration of the polarization sensor according to the first embodiment. As shown in FIG. 8(A), the polarization sensor 33 includes a microlens array 331, a polarizing plate array 332, and a pixel array 333. The microlens array 331 includes a plurality of microlenses, the polarizing plate array 332 includes a plurality of polarizers (polarizing plates), and the pixel array 333 includes a plurality of pixels (picture elements). As shown in the figure, the polarizers are mounted such that the polarization angle is changed by 45 degrees for each pixel. FIG. 8(B) shows the polarization angle for each polarizer. As shown in the figure, the polarization sensor 33 can grasp the polarization degrees of 0 degrees, 45 degrees, 90 degrees, and 135 degrees using four pixels.
[0071] Returning to FIG. 2, the polarization image acquisition unit 131 acquires a polarization image captured by such a polarization sensor 33. The polarization image includes signals in four polarization directions. It can also be said that the polarization image is an image of a subject captured through polarization filters having a plurality of different polarization angles. It is widely known that the polarization angle can be obtained from the signals in the four polarization directions obtained from such a polarization sensor 33, and can be obtained by calculation using, for example, the following equations (6) to (8). The signals of the four polarization angles obtained from the polarization sensor 33 are referred to as H0, H45, H90, and H135.
[0072] Q = H0 - H90 (6) U=H45-H135···(7)
[0073] In this case, the polarization angle AoLP can be calculated by the following equation (8).
[0074] AoLP=1 / 2ATAN(U / Q)···(8)
[0075] The polarization angle calculation unit 132 obtains the AoLP by performing calculations using the above-mentioned equations (6) to (8).
[0076] The resizing unit 133 performs resizing so that the number of pixels with the polarization angle is the same as that of the RGB sensor 32 .
[0077] Here, we note that the polarization angle of the light and shadow areas of a subject can be significantly different, even for the same object. Shadows are created when the way light hits an object is different. Therefore, the polarization angle of the shadow area will be different from that of the light-hit area.
[0078] Fig. 9 shows an example of a case where the polarization angle signals obtained by the polarization sensor according to the first embodiment are color-coded. Fig. 9(A) shows the signals of the four polarization angles obtained by the polarization sensor. Fig. 9(B) shows an example of a case where the polarization angle AoLP is calculated from these signals using the above formula (8) and the angle is color-coded. As shown in the figure, it can be seen that the polarization angles of the shadowed portion and the non-shadowed portion are different even if they are the same object.
[0079] Returning to FIG. 2, the process of detecting a final shadow candidate based on the shadow candidate 1 and the information obtained by the polarization sensor 33 will be described.
[0080] The shadow candidate 2 detection unit 152 obtains the shadow candidate 1 from the shadow candidate 1 detection unit 151 and obtains the OBJ_INDEX number from the object region division unit 118. The shadow candidate 2 detection unit 152 also obtains DEGTH from the storage device 21. DEGTH is a threshold value used to determine whether there is a difference in polarization angle. The shadow candidate 2 detection unit 152 calculates the average polarization angle AoLP for the part where the signal of the shadow candidate 1 is 1 from the OBJ_INDEX number of the same number. Specifically, the shadow candidate 2 detection unit 152 may extract the polarization angle AoLP of the part corresponding to the part where the signal of the shadow candidate 1 is 1 from the information after the polarization angle AoLP is calculated by the polarization angle calculation unit 132 and resizing is performed by the resizing unit 133. The shadow candidate 2 detection unit 152 stores the calculated polarization angle AoLP as AoLP_Shadow1. Next, the shadow candidate 2 detection unit 152 calculates the average polarization angle AoLP of the portion of the same OBJ_INDEX number where the signal of the shadow candidate 1 is 0. The shadow candidate 2 detection unit 152 sets the calculated polarization angle AoLP as AoLP_Shadow0.
[0081] Next, the shadow candidate 2 detection unit 152 calculates the difference between the calculated angle difference AoLP_Shadow1 and AoLP_Shadow0. The shadow candidate 2 detection unit 152 also calculates whether the calculated difference is equal to or greater than DEGTH. The shadow candidate 2 detection unit 152 determines that a portion where the calculated difference is equal to or greater than DEGTH is the final shadow portion, and outputs a SHADOW signal as 1. If the calculated difference is equal to or less than DEGTH, the shadow candidate 2 detection unit 152 sets the SHADOW signal to 0. This SHADOW signal exists for each pixel, and coincides with the resolution of the RGB image and the distance image. In other words, information including the SHADOW signal for each pixel can be called a SHADOW image.
[0082] Fig. 10 is a diagram showing an example of a SHADOW image according to the first embodiment. The SHADOW image shown corresponds to Figs. 3, 5, and 7. 1 indicates a shadow portion, and 0 indicates a non-shadow portion. According to this embodiment, by referring to a SHADOW image as shown in the figure, it becomes possible to determine whether or not each pixel in the RGB image and the distance image is a shadow.
[0083] It is preferable that the SHADOW image is stored in association with the RGB image and the distance image. By storing the SHADOW signal in association with the RGB image and the distance image, even in the case of forming a three-dimensional model, it becomes easy to understand which part is the shadow of the original subject by referring to the SHADOW image.
[0084] Note that the RGB image, distance image, and shadow image according to this embodiment may be captured as successive frames and treated as a video.
[0085] [Summary of embodiment 1] According to the embodiment described above, the shadow area detection device 10 includes the image acquisition unit 111 to acquire an image having information of a subject at each coordinate on a two-dimensional plane, includes the area division unit 119 to divide each coordinate of the acquired image into a plurality of areas based on the information of the subject at each coordinate, includes the polarized image acquisition unit 131 to acquire a plurality of polarized images of the subject captured through polarizing filters having a plurality of different polarization angles, and includes the shadow candidate detection unit 150 to detect candidates of coordinates estimated to be shadow areas for each coordinate of the image based on information on the areas of the image divided by the area division unit 119 and the polarized image acquired by the polarized image acquisition unit 131. That is, according to this embodiment, it is possible to detect a portion where the color information of the subject is darkened due to the influence of a shadow. According to this embodiment, it is possible to detect a portion where the color information of the subject is darkened due to the influence of a shadow, so that when a three-dimensional model is to be generated, it is possible to generate a three-dimensional model with reduced influence of a shadow by correcting the color information of the detected shadow area.
[0086] Furthermore, according to the above-described embodiment, the image acquired by the image acquisition unit 111 includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane, and the region division unit 119 divides each coordinate of the RGB image into a plurality of regions based on the color information of each coordinate. That is, according to this embodiment, shadow candidates are detected based on a high-resolution RGB image and a polarized image. Therefore, according to this embodiment, shadow candidates are detected based on a high-resolution RGB image, so that shadow candidates can be detected with high accuracy.
[0087] Furthermore, according to the above-described embodiment, the image acquired by the image acquisition unit 111 includes a distance image having information on the distance to the subject at each coordinate on a two-dimensional plane, and the region division unit 119 divides each coordinate of the distance image into a plurality of regions based on the distance information of each coordinate. That is, according to this embodiment, shadow candidates are detected based on the distance image and the polarized image. The distance image makes it possible to accurately determine whether or not the objects are the same. Therefore, according to this embodiment, shadow candidates can be detected with high accuracy.
[0088] [Embodiment 2 and embodiment 3] Next, a shadow region detection system 1A according to a second embodiment and a third embodiment will be described with reference to Fig. 11 to Fig. 14. In the second and third embodiments, the above-mentioned SHADOW image is used to correct shadow portions in an RGB image and a distance image. In the second embodiment, shadow correction is performed on an RGB image, and in the third embodiment, shadow correction is performed on a distance image. Below, matters common to the second and third embodiments will be described.
[0089] 11 is a functional configuration diagram showing an example of the functional configuration of a shadow area detection system according to the second embodiment. An example of the functional configuration of shadow area detection system 1A will be described with reference to the diagram. Shadow area detection system 1A differs from shadow area detection system 1 in that it further includes a shadow correction unit 161. In the description of shadow area detection system 1A, components similar to those of shadow area detection system 1 are denoted by similar reference numerals and description thereof may be omitted.
[0090] The shadow correction unit 161 obtains a SHADOW signal from the shadow candidate 2 detection unit 152, obtains a resized distance image from the resizing unit 113, and obtains an RGB signal from the image acquisition unit 111. The shadow correction unit 161 corrects the shadow portion by replacing at least one of the pixel value or distance value of the coordinates estimated as a shadow candidate by the shadow candidate detection unit 150 with at least one of the pixel value or distance value of the coordinates not estimated as a shadow candidate. Specifically, the shadow correction unit 161 obtains, for each pixel, how many pixels away from a non-shadow portion having the same OBJ_INDEX number in both the horizontal and vertical directions based on the obtained information. Specifically, when there is a pixel in a shadow portion with a SHADOW signal of 1 in either the left or right horizontal direction that has the same OBJ_INDEX number, the shadow correction unit 161 calculates how many pixels away it is in the horizontal direction, and sets the calculated value as SHADOW_HDISTANCE. At this time, a sign is attached so that if it is on the right, it is positive, and if it is on the left, it is negative, so that the left and right directions can be determined. If there are objects with the same OBJ_INDEX number in the horizontal direction, the number of the closest object is entered into SHADOW_HDISTANCE. If there are no objects with the same OBJ_INDEX number in the horizontal direction, SHADOW_HDISTANCE is set to 0.
[0091] Furthermore, if there is a pixel in the shadow area with a SHADOW signal of 1 that has the same OBJ_INDEX number either above or below the pixel in the vertical direction, the shadow correction unit 161 calculates how many pixels away it is vertically, and sets the calculated value as SHADOW_VDISTANCE. At this time, to indicate the up and down direction, a sign is assigned so that if it is below, it is positive, and if it is above, it is negative. If there are pixels with the same OBJ_INDEX number in the up and down or vertical direction, the number of the closest distance is entered into SHADOW_VDISTANCE. Furthermore, if there is no pixel with the same OBJ_INDEX number in the up and down direction, SHADOW_VDISTANCE is set to 0.
[0092] Fig. 12 is a diagram for explaining the shadow correction process by the shadow correction unit according to the second embodiment. Fig. 12(A) shows an example of SHADOW_HDISTANCE calculated according to the above method. Fig. 12(B) shows the value of SHADOW_VDISTANCE calculated according to the above method.
[0093] 13 is a functional configuration diagram showing an example of the functional configuration of a shadow correction unit according to the second embodiment. With reference to the same figure, an example of the functional configuration of the shadow correction unit 161 will be described. The shadow correction unit 161 includes a shadow distance calculation unit 1611, a low-pass filter 1612, a subtraction unit 1613, a substitution unit 1614, a low-pass filter 1615, and an addition unit 1616.
[0094] The shadow distance calculation unit 1611 calculates SHADOW_HDISTANCE and SHADOW_VDISTANCE by the above-mentioned method. The shadow distance calculation unit 1611 outputs the calculated values to the low-pass filter 1612, the substitution unit 1614, and the low-pass filter 1615.
[0095] The replacement unit 1614 replaces the absolute values of SHADOW_HDISTANCE and SHADOW_VDISTANCE with RGB signals or distance signals for non-shadow parts of the distance pixel count in the direction of the closer distance. Here, the direction is determined by the signs of SHADOW_HDISTANCE and SHADOW_VDISTANCE. However, if either SHADOW_HDISTANCE or SHADOW_VDISTANCE contains 0, the replacement unit 1614 replaces the non-shadow part of the distance with the RGB signals or distance signals for the non-zero direction.
[0096] The low-pass filter 1615 filters the output of the substitution unit 1614. The low-pass filter 1615 changes the degree of filtering depending on the values of SHADOW_HDISTANCE and SHADOW_VDISTANCE. The longer the distance, the more the low-pass filter 1615 filters, making the image smoother.
[0097] 14 is a functional configuration diagram showing an example of the functional configuration of the LPF according to the second embodiment. With reference to the same figure, an example of the functional configuration of the low-pass filter LPF will be described. The low-pass filter 1612 and the low-pass filter 1615 each have the functional configuration shown in the figure, and may have the same functional configuration. Hereinafter, when the low-pass filter 1612 and the low-pass filter 1615 are not distinguished from each other, they may be referred to as the low-pass filter LPF.
[0098] The low-pass filter LPF includes 3TAP horizontal LPF 411 to 3TAP horizontal LPF 414, HSEL 42, 3TAP vertical LPF 431 to 3TAP horizontal LPF 434, and VSEL 44. The low-pass filter LPF first performs horizontal LPF processing on the RGB signal or the distance signal. The HSEL 42 switches between a through state where the signal does not pass through the 3TAP horizontal LPF, a state where the signal passes through once, a state where the signal passes twice, a state where the signal passes three times, and a state where the signal passes four times, based on the absolute value of SHADOW_HDISTNACE. Specifically, the HSEL 42 may switch between a through state when SHADOW_HDISTANCE is 0, a state where the signal passes through once when SHADOW_HDISTANCE is 1, a state where the signal passes twice when SHADOW_HDISTANCE is 2, and a state where the signal passes three times when SHADOW_HDISTANCE is 3. The HSEL 42 may switch between a four-times-pass state when SHADOW_HDISTANCE is 4 or more.
[0099] Each of the 3TAP horizontal LPFs 411 to 414 may be as shown in the following formula (9).
[0100] H3TAPLPF=0.25Z^(-1)+0.5Z^0+0.25Z^1···(9) Note that Z is the scan time for one pixel.
[0101] The same processing as for the horizontal is performed for the vertical. Specifically, VSL44 switches to one of a through state where the 3TAP vertical LPF is not passed through, a state where the 3TAP vertical LPF is passed once, a state where the 3TAP vertical LPF is passed twice, a state where the 3TAP vertical LPF is passed three times, and a state where the 3TAP vertical LPF is passed four times, based on the absolute value of SHADOW_VDISTNACE. Specifically, VSEL44 may switch to a through state when SHADOW_VDISTANCE is 0, a state where the 3TAP vertical LPF is passed once when SHADOW_VDISTANCE is 1, a state where the 3TAP vertical LPF is passed twice when SHADOW_VDISTANCE is 2, and a state where the 3TAP vertical LPF is passed three times when SHADOW_VDISTANCE is 3. Note that VSEL44 may switch to a state where the 3TAP vertical LPF is passed four times when SHADOW_VDISTANCE is 4 or more.
[0102] Each of the 3TAP vertical LPF 431 to the 3TAP horizontal LPF 434 may be as shown in the following formula (10).
[0103] V3TAPLPF=0.25Z^(-1)+0.5Z^0+0.25Z^1···(10) Note that Z is the scan time for one line.
[0104] The low-pass filter 1615 outputs the signal that has been LPF-processed in this manner as SHADOW_LPF.
[0105] Shadow correction unit 161 may include a low-pass filter 1612, which may apply a similar LPF to the RGB signal or distance signal in an unreplaced state (before passing through replacement unit 1614). The signal that has passed through low-pass filter 1612 is subtracted by subtraction unit 1613 from the RGB signal or distance signal before passing through low-pass filter 1612, thereby making it possible to extract a high-frequency signal. The subtracted signal is added by addition unit 1616 to SHADOW_LPF output from low-pass filter 1615, and output. By adding the high-frequency components extracted from the original signal in this way, it is possible to prevent a loss of detail even when shadow correction is performed.
[0106] The shadow correction unit 161 performs shadow correction by the above-mentioned processing, and stores the RGB signal or distance signal with reduced influence of the shadow in the storage device 22. The correction processing performed by the shadow correction unit 161 may be a correction processing of either the RGB value or the distance value. Since the distance signal obtained from the distance measurement sensor 31 tends to not provide accurate distance accuracy in dark areas such as shadows, correcting only the distance value (without correcting the RGB image) is effective for the purpose of increasing the distance accuracy in the shadow area.
[0107] Note that the RGB image, distance image, and shadow image according to this embodiment may be captured as successive frames and treated as a video.
[0108] [Summary of the second and third embodiments] According to the embodiment described above, the shadow area detection device 10A replaces pixel values of coordinates estimated as shadow candidates with values based on pixel values of coordinates not estimated as shadow candidates by the shadow candidate detection unit 150. Therefore, according to this embodiment, it is possible to detect parts of the subject where color information is darkened due to the influence of a shadow, and to correct the parts based on the RGB values of non-shadow parts. Furthermore, according to this embodiment, it is possible to correct the distance value of a shadow part based on the distance value of a non-shadow part.
[0109] [Embodiment 4] Next, a fourth embodiment will be described with reference to Figs. 15 and 16. In this embodiment, an imaging device including the shadow area detection system according to the first to third embodiments will be described. Here, when the RGB sensor, the polarization sensor, and the TOF sensor are photographed with the same lens on the same optical axis, there is no misalignment between the pixels. By adopting such a configuration, more accurate shadow detection and shadow correction are possible. In this embodiment, an example will be described in which the RGB sensor, the polarization sensor, and the TOF sensor are arranged on the same optical axis using a prism.
[0110] 15 is a cross-sectional view showing a first arrangement example of an imaging device including a shadow area detection system according to the third embodiment. With reference to the figure, an imaging device 51, which is the first arrangement example, will be described. The imaging device 51 includes a depolarizing plate 511, a polarization sensor 512, a non-polarizing half mirror 513, a visible light / infrared separation dichroic coat 514, a ToF sensor 515, and an RGB sensor 516.
[0111] As shown in the figure, the polarization sensor 512 is disposed on the first exit surface that is not affected by the change in polarization caused by the visible light / infrared separation dichroic coat 514. Furthermore, by disposing a depolarizing plate 511 on the first exit surface, the change in the polarization state caused by total reflection in the first prism is eliminated. Furthermore, a non-polarizing half mirror 513 is used as the half mirror. Specifically, the non-polarizing half mirror 513 may be a metal film or a dielectric film. The visible light / infrared separation dichroic coat 514 separates visible light and near-infrared light. As a result of the separation, a ToF sensor 515 is disposed on the third exit surface that transmits the near-infrared light, and an RGB sensor 516 is disposed on the second exit surface that transmits the visible light.
[0112] 16 is a cross-sectional view showing a second arrangement example of an imaging device including a shadow area detection system according to the third embodiment. With reference to the figure, an imaging device 52, which is the second arrangement example, will be described. The imaging device 52 includes a non-polarized visible light / infrared separation dichroic coat 521, a ToF sensor 522, a non-polarized half mirror 523, a polarization sensor 524, and an RGB sensor 525.
[0113] As shown in the figure, in the imaging device 52, the polarization sensor 524 is disposed on the third exit surface that is not affected by the total reflection of the first prism. As the half mirror, a non-polarizing half mirror 523 is used. Specifically, the non-polarizing half mirror 523 may be a metal film or a dielectric film. For the non-polarized visible light / infrared separation dichroic coat 521, a non-polarized dichroic coat that suppresses phase difference due to wavelength is used. A ToF sensor 522 is disposed on the first exit surface through which the near-infrared light reflected by the non-polarized visible light / infrared separation dichroic coat 521 passes, and an RGB sensor 525 is disposed on the second exit surface through which the visible light passes.
[0114] [Summary of the fourth embodiment] According to the above-described embodiment, by using the above-described prism configuration, when configuring an imaging device equipped with the shadow area detection system according to the above-described embodiment 1 to embodiment 3, the RGB sensor, the polarization sensor, and the ToF sensor can be photographed with the same lens and the same optical axis, respectively. Therefore, it is possible to reduce the deviation of each pixel, and more accurate shadow detection and shadow correction can be performed.
[0115] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, reading the program recorded on the recording medium into a computer system, and executing it. Note that the "computer system" referred to here includes hardware such as the OS and peripheral devices.
[0116] In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.
[0117] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate. [Explanation of symbols]
[0118] 1...shadow area detection system, 10...shadow area detection device, 111...image acquisition section, 113...resizing section, 115...hue / brightness calculation section, 116...distance area division section, 117...color area division section, 118...object area division section, 119...area division section, 131...polarized image acquisition section, 132...polarization angle calculation section, 133...resizing section, 150...shadow candidate detection section, 151...shadow candidate 1 detection section, 152...shadow candidate 2 detection section, 21...storage device, 22...storage device, 31...distance measurement sensor, 32...RGB sensor, 33...polarization sensor, 161 shadow correction section, 1611...shadow distance calculation section, 1612...low-pass filter, 1613...subtraction section, 1614...substitution section, 1615...low-pass filter, 1616...addition section, 51...imaging device, 52...imaging device
Claims
1. an image acquisition unit that acquires an image having information of a subject at each coordinate on a two-dimensional plane; a region dividing unit that divides each coordinate of the acquired image into a plurality of regions based on information of the subject that each coordinate has; a polarization image acquisition unit that acquires a plurality of polarization images of the subject through a plurality of polarization filters having different polarization angles; a shadow candidate detection unit that detects candidates of coordinates estimated to be a shadow portion for each coordinate of the image based on information on the image region divided by the region division unit and the polarized image; A shadow area detection device comprising:
2. the image acquired by the image acquisition unit includes an RGB image having information about the color of the subject at each coordinate on a two-dimensional plane; The region dividing unit further includes a color region dividing unit that divides each coordinate of the RGB image into a plurality of regions based on color information that each coordinate has. The shadow region detection device according to claim 1 .
3. The image processing apparatus further includes a shadow correction unit that replaces pixel values of coordinates estimated as a shadow candidate by the shadow candidate detection unit with values based on pixel values of coordinates that are not estimated as a shadow candidate. The shadow region detection device according to claim 2 .
4. the image acquired by the image acquisition unit includes a distance image having information about a distance to the subject at each coordinate on a two-dimensional plane; The region dividing unit further includes a distance region dividing unit that divides each coordinate of the distance image into a plurality of regions based on distance information that each coordinate has. The shadow region detection device according to claim 1 .
5. an image acquisition step of acquiring an image having information of a subject at each coordinate on a two-dimensional plane; a region dividing step of dividing each coordinate of the acquired image into a plurality of regions based on information of the subject held by each coordinate; a polarization image acquisition step of acquiring a plurality of polarization images of the subject through a plurality of polarization filters each having a different polarization angle; a shadow candidate detection step of detecting candidates of coordinates estimated to be a shadow portion for each coordinate of the image based on information on the image regions divided by the region division step and the acquired polarized image; The shadow region detection method has the following structure.
Citation Information
Patent Citations
Imaging device, image processing device and image processing method
WO2016136086A1
