IMAGE CORRECTION DEVICE, IMAGE CORRECTION METHOD, AND IMAGE CORRECTION PROGRAM

Abstract

In the present invention, an input processing unit: receives an image and a three-dimensional point group consisting of three-dimensional points which include the strength of a reflection on the surface of an object for which a relationship between at least an imaging position and a measurement position has been derived in advance; and derives a pixel position on the image corresponding to each of the three-dimensional points in the three-dimensional point group. A shadow region estimation unit: clusters the pixels of the image on the basis of the pixel value and the pixel position; derives an average value for quantified color information and an average reflection strength, for each cluster; and estimates a shadow region by comparing, between the clusters, the average reflection strength and the average value for the color information. A shadow correction unit corrects the pixel values of the shadow region on the basis of the estimated shadow region and the image.

Description

[Technical field]

[0001] The disclosed technology relates to an image correction device, an image correction method, and an image correction program. [Background technology]

[0002] Non-Patent Document 1 discloses a technology that maps the reflection intensity measured by LiDAR onto an image, extracts the shadow boundary area based on the reflection intensity gradient information and color gradient information, derives the average value of the quantified color information for sunlight and shadow, and detects shadows by graph cutting. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] M. Guislain, J. Digne, R. Chaine, D. Kudelski and P. Lefebvre-Albaret, "Detecting and Correcting Shadows in Urban Point Clouds and Image Collections," 2016 Fourth International Conference on 3D Vision (3DV), 2016. Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Non-Patent Document 1, shadow boundary information cannot be directly used in shadow detection. Therefore, when a material with high surface reflectance exists in the shadow, there is a possibility that the shadow will not be detected. Also, there is a possibility that an object with low reflectance existing in the sun will be detected as a shadow. Also, if a measurement device with a large error in reflection intensity is used for measurement, the shadow boundary cannot be detected correctly. Also, there is a possibility that the shadow boundary will become unnatural when the shadow is corrected.

[0005] The disclosed technology has been made in consideration of the above points, and aims to provide an image correction device, an image correction method, and an image correction program that can accurately estimate shadow areas and correct images. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an image correction device that includes an input processing unit that receives an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on the surface of an object, the relationship between at least a shooting position and a measurement position being determined in advance, and determines pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud, a shadow region estimation unit that clusters the pixels of the image based on pixel values ​​and pixel positions, determines an average reflection intensity and an average value of quantified color information for each cluster, and compares the average reflection intensity and the average values ​​of the color information between the clusters to estimate a shadow region, and a shadow correction unit that corrects the pixel values ​​of the shadow region based on the estimated shadow region and the image.

[0007] A second aspect of the present disclosure is an image correction method, in which an input processing unit receives an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on a surface of an object, the relationship between at least a shooting position and a measurement position being determined in advance, and determines pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud, a shadow region estimation unit clusters the pixels of the image based on pixel values ​​and pixel positions, and determines an average reflection intensity and an average value of quantified color information for each cluster, and estimates a shadow region by comparing the average reflection intensity and the average value of the color information between the clusters, and a shadow correction unit corrects the pixel values ​​of the shadow region based on the estimated shadow region and the image.

[0008] A third aspect of the present disclosure is an image correction program for causing a computer to function as the image correction device of the first aspect. Effect of the Invention

[0009] According to the disclosed technology, it is possible to accurately estimate a shadow region and correct an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a schematic block diagram of an example of a computer that functions as the image correction apparatus of the present embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of measurement points by a LiDAR sensor and a scene captured by a camera. [Diagram 3] 1 is a block diagram showing a configuration of an image correction device according to an embodiment of the present invention. [Figure 4] 2 is a block diagram showing a configuration of an intensity correction unit of the image correction device of the present embodiment. FIG. [Diagram 5] FIG. 13 is a diagram showing an example of a result of projecting a three-dimensional point cloud onto an image. [Figure 6] 2 is a block diagram showing a configuration of a shadow region estimation unit of the image correction device of the present embodiment. FIG. [Figure 7] FIG. 13 is a diagram showing an example of an image. [Figure 8] FIG. 13 is a diagram showing an example of a result of clustering pixels of an image. [Figure 9] FIG. 2 is a diagram for explaining a boundary in an image. [Figure 10] FIG. 13 is a diagram for explaining a boundary in a reflection intensity map. [Figure 11] 5 is a flowchart showing an image correction routine of the image correction device of the present embodiment. [Figure 12] 5 is a flowchart showing a process for generating a reflection intensity map in the image correction device of the present embodiment. [Figure 13] 5 is a flowchart showing the flow of processing for estimating a shadow region in the image correction device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0012] <Outline of this embodiment> Conventional technology does not directly utilize shadow boundary information, which can result in the shadow of a material with high surface reflectance in the shade being missed, or a material with low reflectance in the sun being detected as a shadow.

[0013] In this embodiment, the shadow boundary is extracted based on a cluster graph based on the reflection intensity measured by the LiDAR sensor and the brightness value of the image captured by the camera, thereby suppressing the effect of differences in reflectance between materials and accurately estimating the shadow area.

[0014] Furthermore, in the conventional technology, when using observation results from a measuring instrument with a large error in reflection intensity, the boundary of the shadow area may not be detected correctly.

[0015] In this embodiment, by correcting the reflection intensity measured by the LiDAR sensor, the boundary of the shadow area is estimated with high accuracy even when observation data from a measuring instrument with large errors in reflection intensity is used.

[0016] Furthermore, in the conventional technology, when correcting pixel values ​​in a shadow region, the color information of the shadow region may become unnatural compared to the sunny region.

[0017] In this embodiment, based on information about the shadow region that has been accurately estimated, pixel values ​​in the shadow region are corrected so that color information in the shadow region becomes more natural compared to the sunny region.

[0018] <Configuration of the image correction device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of an image correction device 10 of this embodiment.

[0019] 1, an image correction device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0020] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an image correction program for correcting an image using a three-dimensional point cloud. The image correction program may be one program, or may be a group of programs composed of multiple programs or modules.

[0021] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0022] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs including a captured image I and a three-dimensional point group P on the surface of an object for which at least the relationship between the capture position and the measurement position has been obtained in advance. For example, the input unit 15 receives a three-dimensional point group P measured by a LiDAR sensor 50 and an image I captured by a camera 52 in a capture scene, as shown in FIG. 2. The relationship between the capture position of the camera 52 and the measurement position of the LiDAR sensor 50 is obtained in advance. FIG. 2 shows an example in which a three-dimensional point group indicated by a white circle on the surface of an object included in the capture scene of the camera 52 is measured by the LiDAR sensor 50.

[0023] The image I is a distortion-corrected RGB or grayscale image. The three-dimensional point cloud P is a collection of three-dimensional points measured by the LiDAR sensor 50. Each three-dimensional point is a four-dimensional vector consisting of a three-dimensional position and a one-dimensional reflection intensity, and when the three-dimensional point cloud P includes N three-dimensional points, the three-dimensional point cloud P is a collection of four-dimensional vectors having N elements.

[0024] In addition, the input unit 15 receives as input the internal parameters K of the camera 52, the projection matrix R between the camera 52 and the LiDAR sensor 50, and the translation vector L between the camera 52 and the LiDAR sensor 50.

[0025] The internal parameters K of the camera 52 are a 3 × 3 camera internal parameter matrix. The projection matrix R between the camera 52 and the LiDAR sensor 50 is a 3 × 3 rotation matrix. The translation vector L between the camera 52 and the LiDAR sensor 50 is a three-dimensional vector.

[0026] The display unit 16 is, for example, a liquid crystal display, and displays various information including an image corrected using the three-dimensional point cloud P measured by the LiDAR sensor 50. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0027] Specifically, the display unit 16 displays a reflection intensity map that indicates reflection intensity as a pixel value of each pixel, a shadow area mask that indicates a shadow area on the image I, and a corrected image in which the pixel values ​​(color information) of the shadow area are corrected.

[0028] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0029] Next, a description will be given of the functional configuration of the image correction device 10. FIG.

[0030] As shown in FIG. 3, the image correction device 10 functionally comprises a storage unit 20, an intensity correction unit 22, a shadow region estimation unit 24, and a shadow correction unit .

[0031] The storage unit 20 stores the input three-dimensional point group P measured by the LiDAR sensor 50 and the image I captured by the camera 52. The storage unit 20 also stores the input internal parameters K of the camera 52, the projection matrix R between the camera 52 and the LiDAR sensor 50, and the translation vector L between the camera 52 and the LiDAR sensor 50.

[0032] The intensity correction unit 22 generates a reflection intensity map in which a reflection intensity is assigned to each pixel of the image I, based on the three-dimensional point group P measured by the LiDAR sensor 50, the image I captured by the camera 52, an internal parameter K of the camera 52, a projection matrix R between the camera 52 and the LiDAR sensor 50, and a translation vector L between the camera 52 and the LiDAR sensor 50. Furthermore, the intensity correction unit 22 corrects the generated reflection intensity map, based on the image I.

[0033] Specifically, the intensity correction unit 22 includes an input processing unit 32, a tensor calculation unit 34, and a map generation unit 36, as shown in FIG.

[0034] The input processing unit 32 projects each of the three-dimensional points of the three-dimensional point group P onto the image I based on the image I, the three-dimensional point group P, the internal parameters K, the projection matrix R, and the translation vector L, and obtains the pixel position on the image I corresponding to each three-dimensional point (see FIG. 5). FIG. 5 shows an example in which each of the three-dimensional points on the surface of an object represented by a dot is projected onto the image I, and the color density of the dot represents the magnitude of the reflection intensity. Also, an example is shown in which the LiDAR sensor 50 has multiple elements, three-dimensional points measured by one element are projected to be aligned in a row, and the reflection intensity of one row of three-dimensional points measured by a certain element is relatively large. That is, an example is shown in which different reflection intensities are measured for each element even on an object surface where the reflection intensity should originally be the same value.

[0035] The calculation performed by the input processing unit 32 when projecting the three-dimensional point group P onto the image I is the same as the method described in Non-Patent Document 2.

[0036] [Non-Patent Document 2] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision Meets Robotics: The KITTI Dataset,” The International Journal of Robotics Research 32, 2013.

[0037] Specifically, each 3D point of the 3D point group P is projected onto image I, and the 3D points projected outside the area of ​​image I are removed from the 3D point group P. A set Q consisting of elements that are combinations of pixel positions and reflection intensities on image I corresponding to each 3D point of the 3D point group P after removal is obtained.

[0038] For example, let P_1 be the 3D point group P from which only the positional information has been extracted. P_1 is a collection of 3D vectors with N elements. For each of the 3D points included in P_1, a projection matrix R and a translation vector L are applied according to the following formula to obtain P_2. P_2 is a collection of N 3D vectors.

[0039] P_2=R·P_1+L

[0040] The 3D point group P_2 is projected onto the image I of the camera 52 using the internal parameter K, those projected outside the region of the image I are removed, and for 3D points with overlapping projected positions, those closest to the camera 52 are used. This calculation obtains a set Q of elements q consisting of (x coordinate in image I, y coordinate in image I, reflection intensity).

[0041] Note that the correspondence between each element of set Q and each 3D point of the pre-conversion 3D point group P is maintained. Each element of set Q is mapped onto image I to generate a pre-correction reflection intensity map.

[0042] Based on the image I, the tensor calculation unit 34 calculates an anisotropic diffusion tensor T for weighting the smoothing term based on the image gradient.

[0043] The calculation of this anisotropic diffusion tensor is similar to the method described in Non-Patent Document 3.

[0044] [Non-Patent Document 3] D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether and H. Bischof, "Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation," 2013 IEEE International Conference on Computer Vision, 2013, pp. 993-1000.

[0045] Specifically, an anisotropic diffusion tensor T is calculated from image I to weight the smoothing term for the energy function (described later) that smooths based on the image gradient. This anisotropic diffusion tensor T has dimensions of 2 x 2 for each pixel. The gradient N_1 of image I is calculated by differentiating image I in both the x and y directions. The element of the calculated gradient N_1 at pixel (x, y) is a two-dimensional vector (x-direction differential value, y-direction differential value). Since the gradient cannot be defined at the edges of the image, the value of the nearest point is used as the gradient of the undefined region.

[0046] At each pixel of gradient N_1, the gradient N_2 is normalized so that the sum of the squares of the x-direction differential value and the y-direction differential value is 1. Also, the set of vectors calculated by calculating the perpendicular unit vectors at each pixel of gradient N_2 is N_3. The anisotropic diffusion tensor T is calculated from the gradients N_1, N_2, and N_3 according to the following formula.

[0047] T = exp(-a|N_1| b )((N_2)(N_2)')+(N_3)(N_3)'

[0048] In the above formula, A' is the transposed matrix of the matrix A. In the map generating unit 36, the reflection intensity map is corrected by this anisotropic diffusion tensor T with reference to the gradient of the image I.

[0049] The map generator 36 corrects the reflection intensity map, in which a reflection intensity is assigned to each pixel of the image I, based on the difference between the reflection intensity and the reflection intensity of a three-dimensional point corresponding to the pixel position on the image I, and the anisotropic diffusion tensor T.

[0050] Specifically, a data term is calculated for each element q = (x_q, y_q, i_q) of set Q. The data term can be evaluated only from the difference between the measured reflection intensity and the corrected reflection intensity, and evaluates the likelihood of each element g = (x_g, y_g, i_g) of the corrected reflection intensity map C. The cost c_qg for pixel (x, y) is calculated as follows:

[0051] c_qg = w_qg Distance(i_q,i_g)

[0052] The distance is the distance between the reflected intensity \(i_q\) assigned to the pixel \((x, y)\) and the corrected reflected intensity \(i_g\). The distance at this time is, for example, the l1 distance, the l2 distance, or the Huber distance. \(w_qg\) is a weight. If the reflected intensity of the measured 3D point is not projected onto the pixel \((x, y)\), then \(w_qg = 0\). On the other hand, if the reflected intensity of the measured 3D point is projected onto the pixel \((x, y)\), it becomes a value determined according to the degree of smoothing by the smoothing term of the energy function described later (\(0 < w_qg\)). That is, this cost \(c_qg\) means the consistency with the corrected reflected intensity \(i_q\) when the reflected intensity \(i_g\) of the measured 3D point is assigned to the pixel \((x, y)\).

[0053] Then, by the Variational method for obtaining a function that minimizes the functional, a corrected reflected intensity map \(C\) is generated and output. In the present embodiment, the reflected intensity \(i_g\) of each pixel is obtained so as to minimize the following energy function \(E_V\) by the Variational method.

[0054] JPEG0007643590000001.jpg2193

[0055] However, norm is, for example, the l1 distance, the l2 distance, or the Huber distance, and truncation may be performed. \(T\) is an anisotropic diffusion tensor. The minimization of the energy function \(E_V\) can be implemented by the first order primal dual algorithm described in Non-Patent Document 4. By minimizing the energy function \(E_V\), the corrected reflected intensity map \(C\) is obtained.

[0056] [Non-Patent Document 4] Chambolle, A., Pock, T. A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging. J Math Imaging Vis 40, 120 - 145 (2011).

[0057] The shadow region estimation unit 24 clusters the pixels of image I based on their pixel values ​​and pixel positions, integrates the clusters using the corrected reflection intensity map, and calculates the average reflection intensity, average pixel value, and average value of color information for each cluster. The shadow region estimation unit 24 compares the average reflection intensity and the average value of color information between clusters to estimate a shadow region, and outputs a shadow region mask representing the shadow region.

[0058] Specifically, the shadow region estimation unit 24 includes a clustering unit 42, a shadow boundary estimation unit 44, a cost assignment unit 46, and a mask generation unit 48, as shown in FIG.

[0059] The clustering unit 42 clusters the pixels of the image I based on the pixel values, pixel positions, and corrected reflection intensity map of the image I, and calculates the average reflection intensity, average pixel value, and average value of color information for each cluster.

[0060] Specifically, clustering is performed to group together pixels that are close in distance and color space, the clusters are integrated based on reflectance intensity, and the average reflectance intensity, average pixel value, and average color information are calculated for each cluster, to construct a cluster graph G1.

[0061] For example, for image I shown in Fig. 7, clustering is performed to group together pixels that are close in distance space and color space (see Fig. 8). Fig. 8 shows an example in which an image of 50 pixels x 80 pixels is divided into seven clusters.

[0062] There are various clustering methods, and as an example, the SLIC method described in Non-Patent Document 5 may be used.

[0063] Each node n in the cluster graph represents a cluster, where n = (b_n, i_n, lab_n), where b_n is the average value of color information in the cluster, i_n is the average reflectance intensity in the cluster, and lab_n is the average Lab value (three-dimensional) in the cluster. Edges are defined between adjacent clusters. In other words, nodes representing adjacent clusters are connected by edges.

[0064] [Non-Patent Document 5] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua and S. Susstrunk, “SLIC Superpixels Compared to State-of-the-Art Superpixel Methods,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012.

[0065] The shadow boundary estimation unit 44 compares the average reflection intensity and the average value of color information between adjacent clusters, and estimates that the boundary between adjacent clusters is the boundary of a shadow region when the difference in the average color information is large and the difference in the average reflection intensity is small (see Figs. 9 and 10).The shadow boundary estimation unit 44 then assigns a sunny label to the cluster with higher brightness among the adjacent clusters sandwiching the boundary estimated to be the boundary of the shadow region, assigns a shadow label to the cluster with lower brightness, and assigns an unknown label to the cluster that does not include the boundary estimated to be the boundary of the shadow region.

[0066] FIG. 9 shows an example of an image in which the luminance is reduced in a shadow region because the incident light is blocked. FIG. 10 shows an example of a reflection intensity map in which the reflection intensity measured by the LiDAR sensor 50 depends on the reflection intensity of the object surface, and therefore does not depend on whether the region is a shadow or not. At boundary 1, both the luminance and reflection intensity change, so it is determined that it is not a shadow boundary. At boundary 2, the luminance changes, but the reflection intensity does not change, so it is determined that it is a shadow boundary.

[0067] Specifically, from the edges of the cluster graph G1, edges between adjacent clusters where the difference in the average value of color information is large and the difference in the average reflection intensity is small are extracted as edges that are likely to be the boundary of a shadow region, and a set of extracted edges S is obtained. Note that color difference (CIEDE2000, etc.) may be calculated as the difference in color information.

[0068] Then, a label l_n is assigned to a node n connected to an edge belonging to the edge set S as follows.

[0069] A sunshine label (l_n=-1) is assigned to node n with a higher average luminance, indicating that it is likely to be in the sun. A shadow label (l_n=1) is assigned to node n with a lower average luminance, indicating that it is likely to be in the shadow.

[0070] In addition, an unknown label (l_n=0) is assigned to a node n that is not connected to an edge belonging to the edge set S, indicating that it is uncertain whether the node is in the sun or in the shadow. The value of the label l_n can be any ternary value.

[0071] The graph obtained by adding the above labels to each node n of the cluster graph G1 is called cluster graph G2.

[0072] The cost assigning unit 46 adds, in the cluster graph, source edges connecting each of the nodes representing each of the clusters to the source node, and target edges connecting each of the nodes representing each of the clusters to the target node.

[0073] Then, the cost assignment unit 46 assigns a low edge cost to the source side edge of the cluster to which the sunny label has been assigned and a high edge cost to the target side edge, and assigns a high edge cost to the source side edge of the cluster to which the shadow label has been assigned and a low edge cost to the target side edge.

[0074] Further, the cost assigning unit 46 assigns an edge cost corresponding to the distance between the color information of the cluster and the average value of the color information of the clusters with the sun label to the source-side edge of the cluster with the unknown label, and assigns an edge cost corresponding to the distance between the color information of the cluster and the average value of the color information of the clusters with the shadow label to the target-side edge.

[0075] Specifically, the cost assigning unit 46 adds a source node and a target node to the cluster graph G2, and connects each node n = (b_n, i_n, l_n, lab_n) to each of the source node and the target node. Then, the cost assigning unit 46 calculates, as data items, the edge cost of the source-side edge connecting to the source node and the edge cost of the target-side edge connecting to the target node. The edge cost of the source-side edge is a value for evaluating the sun-like degree, and the edge cost of the target-side edge is a value for evaluating the shadow-like degree.

[0076] For example, the average value Ll of the color information is calculated from the node set with the sun label (l_n = -1), and the average value Ls of the color information is calculated from the node set with the shadow label (l_n = 1).

[0077] Also, edge costs corresponding to the label l_n are assigned to each of the source-side edge and the target-side edge of the node n as follows.

[0078] For the node with the shadow label (l_n = +1), let the edge cost of the source-side edge be c1 and the edge cost of the target-side edge be c2. However, c1 < c2, and c1 and c2 are constants defined in advance.

[0079] Also, for the node with the sun label (l_n = -1), let the edge cost of the source-side edge be c2 and the edge cost of the target-side edge be c1.

[0080] For nodes with unknown label (l_n=0), the edge cost of the source edge is Distance(b_n,Ls) and the edge cost of the target edge is Distance(b_n,Ll), where Distance(i,j) is the distance between i and j, typically l1 distance, l2 distance, etc.

[0081] This makes it easier to determine that an area that is highly likely to be a shadow is a shadow, and easier to determine that an area that is highly likely to be a sunny spot is a sunny spot. In an area with an unknown label, the likelihood of it being a shadow is determined based on the brightness value.

[0082] As described above, a source node, a target node, and a source side edge and a target side edge are added to the cluster graph G2, and edge costs are assigned to the resulting graph, which is called cluster graph G3.

[0083] The mask generation unit 48 estimates a shadow region by determining whether each of the clusters is a shadow region based on the edge costs of the source side edges and the edge costs of the target side edges of each of the clusters, and generates a shadow region mask representing the estimated shadow region.

[0084] Specifically, we estimate the shadow region by graph cut by adding a smoothing term to each edge of the cluster graph G3. Here, the smoothing term indicates the edge cost connecting adjacent clusters.

[0085] The mask generating unit 48 may use any binarization image processing method that takes into account the above-mentioned data terms, but the preferred image processing method is the graph cut segmentation method. In this case, the brightness distance Distance(b_n1, b_n2) between adjacent nodes n1=(b_n1, i_n1, l_n1, lab_n1) and n2=(b_n2, i_n2, l_n2, lab_n2) may be used as the smoothing term. The graph cut segmentation method is a well-known image processing method.

[0086] In addition, a label k_n is assigned to every node n in the cluster graph G3 by a binarization method. That is, a label k_n=-1 is assigned to a node n determined to be in the sun, and a label k_n=1 is assigned to a node n determined to be in the shadow.

[0087] Then, a shadow region mask representing a shadow region is generated by assigning -1 or 1 to each pixel corresponding to a cluster represented by each node n in the cluster graph G3 according to the label k_n. Here, the pixel value of the shadow region mask and k_n can be any binary value.

[0088] Moreover, a cluster graph obtained by adding a label k_n to each node of the cluster graph G3 is defined as G4. The mask generating unit 48 outputs the cluster graph G4 and a shadow region mask.

[0089] The shadow correction unit 26 corrects the pixel values ​​of the shadow region based on the estimated shadow region and the image I.

[0090] Specifically, the pixel values ​​of the shadow region are corrected based on the information (l_n, k_n) of each node of the cluster graph G4 and the image I.

[0091] For example, the average Lab value is calculated from the node set with label l_n=-1 and the node set with label l_n=1, and is defined as (lab_l, lab_s).

[0092] Also, perform the following calculation for each node n=(b_n, i_n, l_n, lab_n).

[0093] If the label k_n=-1, the pixel values ​​of each pixel in the cluster are left as they are without correction. If the labels k_n=1, l_n=1, the Lab value (lab_m) of node m connected to node n by an edge belonging to set S is corrected to be the Lab value of each pixel in the cluster.

[0094] Furthermore, when label k_n=1 and label l_n≠1, the Lab values ​​calculated by the following formula are corrected to become the Lab values ​​of each pixel in the cluster.

[0095] Lab value = lab_n*(lab_l / lab_s)

[0096] As described above, the shadow correction unit 26 corrects the Lab values ​​of each pixel in a cluster determined to be a shadow region of image I using the Lab values ​​of an adjacent cluster or the Lab values ​​of a cluster determined to be in the sun. Note that the RGB values ​​of each pixel in a cluster determined to be a shadow region may be corrected using the RGB values ​​of an adjacent cluster or the RGB values ​​of a cluster determined to be in the sun.

[0097] <Function of the image correction device according to this embodiment> Next, the operation of the image correction device 10 will be described.

[0098] 11 is a flowchart showing the flow of image correction processing by the image correction device 10. The CPU 11 reads out an image correction program from the ROM 12 or the storage 14, deploys it in the RAM 13, and executes it to perform the image correction processing. In addition, a three-dimensional point group P measured by the LiDAR sensor 50 and an image I captured by the camera 52 are input to the image correction device 10. In addition, it is assumed that an internal parameter K of the camera 52, a projection matrix R between the camera 52 and the LiDAR sensor 50, and a translation vector L between the camera 52 and the LiDAR sensor 50 are input to the image correction device 10.

[0099] In step S100, the CPU 11, functioning as the intensity correction unit 22, generates a reflection intensity map in which a reflection intensity is assigned to each pixel of the image I, based on the three-dimensional point group P, the image I, the internal parameters K, the projection matrix R, and the translation vector L. The CPU 11, functioning as the intensity correction unit 22, also corrects the generated reflection intensity map, based on the image I.

[0100] In step S102, the CPU 11, functioning as the shadow region estimation unit 24, clusters the pixels of image I based on pixel values ​​and pixel positions, integrates the clusters using the corrected reflection intensity map, and calculates the average reflection intensity, average pixel value, and average value of color information for each cluster. The CPU 11, functioning as the shadow region estimation unit 24, also compares the average reflection intensity and average value of color information between clusters, estimates a shadow region, and outputs a shadow region mask representing the shadow region.

[0101] In step S104, the CPU 11, as the shadow correction unit 26, corrects the pixel values ​​of the shadow area based on the estimated shadow area and image I, displays the reflection intensity map, the shadow area mask, and the corrected image on the display unit 16, and ends the image correction routine.

[0102] The above step S100 is realized by a processing routine shown in FIG.

[0103] In step S110, the CPU 11, as the input processing unit 32, projects each 3D point of the 3D point group P onto the image I based on the image I, the 3D point group P, the internal parameters K, the projection matrix R, and the translation vector L, and performs mapping to determine the pixel positions on the image I corresponding to each 3D point.

[0104] In step S112, the CPU 11, functioning as the tensor calculation unit 34, calculates an anisotropic diffusion tensor T that weights the smoothing term based on the image gradient.

[0105] In step S114, the CPU 11, as the map generation unit 36, generates a corrected reflection intensity map by correcting the reflection intensity map in which a reflection intensity is assigned to each pixel of the image I based on the difference between the reflection intensity and the reflection intensity of the three-dimensional point corresponding to the pixel position on the image I, and based on the anisotropic diffusion tensor T.

[0106] The above step S102 is realized by a processing routine shown in FIG.

[0107] In step S120, the CPU 11, as the clustering unit 42, clusters the pixels of image I based on the pixel values, pixel positions, and reflection intensity map of image I, and calculates the average reflection intensity, average pixel value, and average value of color information for each cluster.

[0108] In step S122, the CPU 11, as the shadow boundary estimation unit 44, compares the average reflection intensity and the average value of the color information between adjacent clusters, and if the difference in the average value of the color information is large and the difference in the average reflection intensity is small, it estimates that the boundary between the adjacent clusters is the boundary of the shadow region.

[0109] In step S124, the CPU 11, as the cost assigning unit 46, adds source-side edges connecting each of the nodes representing each of the clusters to the source node, and target-side edges connecting each of the nodes representing each of the clusters to the target node. Then, the CPU 11, as the cost assigning unit 46, assigns a low edge cost to the source-side edges of the clusters to which the sunny label has been assigned, assigns a high edge cost to the target-side edges, assigns a high edge cost to the source-side edges of the clusters to which the shaded label has been assigned, and assigns a low edge cost to the target-side edges.

[0110] In addition, the CPU 11, as a cost assignment unit 46, assigns a cost to the source side edge of a cluster that has been assigned an unknown label based on the distance between the color information of the cluster and the average value of the color information of clusters that have been assigned a sunny label, and assigns a cost to the target side edge based on the distance between the color information of the cluster and the average value of the color information of clusters that have been assigned a shadow label.

[0111] In step S126, the CPU 11, as the mask generation unit 48, estimates the shadow area by determining whether each of the clusters is a shadow area based on the edge costs of the source side edges and the edge costs of the target side edges of each of the clusters, and generates a shadow area mask.

[0112] As described above, the image correction device according to this embodiment determines the pixel position on the image corresponding to each of the 3D points of the 3D point cloud, clusters the pixels of the image based on the pixel value and pixel position, determines the average value of the average reflection intensity and color information for each cluster, compares the average values ​​of the average reflection intensity and color information between clusters to estimate a shadow region, and corrects the pixel value of the shadow region from the estimated shadow region and the image. This makes it possible to accurately estimate the shadow region and correct the image.

[0113] Furthermore, even if there is a large variation in the reflection intensity measured by the LiDAR sensor, it is possible to estimate shadow areas while reducing the influence of differences in the reflectance of each material by using the reflection intensity, and it is possible to provide consistent color information between shadow areas and sunny areas after correction.

[0114] In addition, by using reflection intensity that is not dependent on the shadows in the image, it is possible to prevent the shadows from being overlooked due to objects with high surface reflectance in the shade, and to prevent objects with low reflectance in the sun from being detected as shadows. This makes it possible to reduce the influence of differences in surface reflectance between objects when estimating shadow regions.

[0115] In addition, since the reflection intensity measured by the LiDAR sensor is corrected, it can be used even in cases where the measurement instrument has a large error in reflection intensity.

[0116] In addition, by extracting edges between adjacent clusters in the cluster graph where the difference in the average value of color information is large and the difference in average reflection intensity is small, pairs of sunny and shadow areas of the same object are detected, and the Lab values ​​of the shadow areas are corrected, a natural image in which the colors of the shadow and sunny areas match can be generated.

[0117] <Modification> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0118] For example, although the case where a 3D point cloud is acquired by measurement using a LiDAR sensor has been described as an example, the present invention is not limited to this. A sensor other than a LiDAR sensor may be used to measure the 3D point cloud.

[0119] In addition, various types of processing that the CPU reads and executes software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electric circuits that are processors having circuit configurations designed exclusively for executing specific processing, such as GPUs (Graphics Processing Units), PLDs (Programmable Logic Devices) whose circuit configurations can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits). In addition, the image correction process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0120] In addition, in each of the above embodiments, the image correction program is described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0121] The following supplementary notes are further disclosed regarding the above embodiment.

[0122] (Additional note 1) An image correction device, comprising: Memory, at least one processor coupled to the memory; Including, The processor, receiving an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on a surface of an object, the relationship between at least a photographing position and a measurement position being determined in advance; determining pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud; The pixels of the image are clustered based on pixel values ​​and pixel positions, and an average reflection intensity and an average value of quantified color information are calculated for each cluster; comparing the average reflectance and the average color information between the clusters to estimate a shadow region; The pixel values ​​of the shadow region are corrected based on the estimated shadow region and the image. The image correction device is configured as follows.

[0123] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform an image correction process, The image correction process includes: receiving an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on a surface of an object, the relationship between at least a photographing position and a measurement position being determined in advance; determining pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud; The pixels of the image are clustered based on pixel values ​​and pixel positions, and an average reflection intensity and an average value of quantified color information are calculated for each cluster; comparing the average reflectance and the average color information between the clusters to estimate a shadow region; The pixel values ​​of the shadow region are corrected based on the estimated shadow region and the image. Non-transitory storage medium. [Explanation of symbols]

[0124] 10. Image correction device 11 CPU 14. Storage 15 Input section 16 Display 20 Memory section 22 Intensity correction section 24 Shadow area estimation section 26 Shadow correction section 32 Input Processing Section 34 Tensor Calculation Unit 36 Map Generation Unit 42 Clustering Department 44 Shadow boundary estimation part 46 Cost Allocation Department 48 Mask Generation Unit 50 LiDAR sensors 52 Camera

Claims

1. an input processing unit that receives an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on a surface of an object, the relationship between at least a shooting position and a measurement position being previously determined, and determines pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud; The pixels of the image are clustered based on pixel values ​​and pixel positions, and an average reflection intensity and an average value of quantified color information are calculated for each cluster; a shadow region estimation unit that estimates a shadow region by comparing the average reflection intensity and the average value of the color information between the clusters; a shadow correction unit that corrects pixel values ​​of the shadow area based on the estimated shadow area and the image; An image correction device comprising:

2. 2. The image correction device according to claim 1, wherein the shadow region estimation unit estimates that the boundary between the adjacent clusters is a boundary of a shadow region when a difference in the average values ​​of the color information between the adjacent clusters is large and a difference in the reflection intensity is small.

3. the shadow region estimation unit assigns a sunshine label to a cluster having a higher brightness among the adjacent clusters sandwiching the boundary estimated as the boundary of the shadow region, assigns a shadow label to a cluster having a lower brightness, and assigns an unknown label to a cluster that does not include the boundary estimated as the boundary of the shadow region; In a graph including nodes representing each of the clusters, a source side edge connecting each of the nodes representing each of the clusters to a source node and a target side edge connecting each of the nodes representing each of the clusters to a target node are provided; A low edge cost is assigned to the source side edge of the cluster to which the Hyuga label is assigned, and a high edge cost is assigned to the target side edge of the cluster; assigning a high edge cost to the source-side edges of the cluster to which the shadow label is assigned and a low edge cost to the target-side edges of the cluster to which the shadow label is assigned; assigning an edge cost to the source side edge of the cluster to which the unknown label is assigned according to the distance between the color information of the cluster and the average value of the color information of the cluster to which the sunny label is assigned, and assigning an edge cost to the target side edge according to the distance between the color information of the cluster and the average value of the color information of the cluster to which the shadow label is assigned; 3. The image correction device according to claim 2, wherein the shadow region is estimated based on the edge costs assigned in the graph.

4. a map generating unit that generates a reflection intensity map in which a reflection intensity is assigned to each pixel of the image based on a difference between the reflection intensity of the pixel and a reflection intensity of a three-dimensional point corresponding to the pixel position on the image; 4. The image correction device according to claim 1, wherein the shadow region estimation section performs clustering of the pixels of the image based on the pixel value, the pixel position, and the reflection intensity map.

5. 4. The image correction device according to claim 3, wherein the shadow region estimation unit estimates the shadow region by determining whether each of the clusters is a shadow region based on the edge costs of the source side edges and the edge costs of the target side edges of each of the clusters.

6. an input processing unit receives an image and a three-dimensional point cloud consisting of three-dimensional points having reflection intensities on a surface of an object for which a relationship between at least a shooting position and a measurement position is previously obtained, and determines pixel positions on the image corresponding to each of the three-dimensional points of the three-dimensional point cloud; a shadow region estimation unit that performs clustering of pixels of the image based on pixel values ​​and pixel positions, and calculates an average reflection intensity and an average value of quantified color information for each cluster; comparing the average reflectance and the average color information between the clusters to estimate a shadow region; A shadow correction unit corrects pixel values ​​of the shadow region based on the estimated shadow region and the image. Image correction methods.

7. An image correction program for causing a computer to function as the image correction device according to any one of claims 1 to 5.

Images