Image processing device and image processing program

Abstract

An image processing device according to one aspect of the present invention comprises a specific frequency laser reflectance calculation unit, an incident light calculation unit, a measurement light calculation unit, and an albedo calculation unit. The specific-frequency laser reflectance calculation unit calculates a specific-frequency laser reflectance, which is a reflectance at the wavelength of specific-frequency laser, from a specific-frequency laser reflection intensity, which is the result of measuring a real-world object with the specific-frequency laser. The incident light calculation unit calculates incident light values, which are the values of white board data at various wavelengths including the wavelength of the specific-frequency laser, from white board data, which is a result of image capture of a white board by a multispectral camera. The measurement light calculation unit calculates, from multispectral camera data, which is a result of image capture of an object by the multispectral camera, measurement values, which are the values of the multispectral camera data at various wavelengths including the wavelength of the specific-frequency laser. The albedo calculation unit calculates an albedo, which is an object-specific reflectance, from the specific-frequency laser reflectance, the incident light value, and the measurement value.

Description

Image processing device and image processing program

[0001] One aspect of the present invention relates to an image processing device and an image processing program.

[0002] In recent years, various computer vision tasks (hereinafter abbreviated as CV tasks) such as image recognition and semantic segmentation have been widely used.

[0003] It is generally known that shadows degrade the performance of detection and estimation in these CV tasks. Therefore, eigenimage decomposition, which decomposes an image into components independent of illuminating light (albedo) and components dependent on illuminating light (shade), has attracted attention. Eigenimage decomposition can be broadly divided into two types: computer graphics (CG)-based and real-world database-based.

[0004] In CG-based eigenimage decomposition, the albedo of a 3D model is defined and then rendered to generate a set of images, albedos, and shades. Therefore, quantitative evaluation of CG-based eigenimage decomposition tasks is easy. However, in practice, eigenimage decomposition must be performed on real-world data.

[0005] Therefore, the other type of eigenimage decomposition of real-world data calculates the albedo and shade of real-world data. In the real world, it is difficult to completely eliminate sunlight conditions from captured images, so it is difficult to obtain correct answers for albedo and shade.

[0006] Therefore, Non-Patent Document 1 proposes a method for quantitatively evaluating eigenimage decomposition by manually performing relative reflection intensity annotation. Specifically, Non-Patent Document 1 uses Poisson sampling to extract any two points in an image, and evaluates the relative reflection intensity information and the confidence level for the annotation. This operation is performed for multiple points in the image to create an annotation for one image. This is then performed for all target images. In addition, multiple people annotate one image, and the accuracy of the evaluation is estimated. The degree of agreement between the annotation and the albedo calculated by eigenimage decomposition is quantitatively evaluated.

[0007] Bell, S., Bala, K., & Snavely, N., (2014), “Intrinsic images in the wild”, ACM Transactions on Graphics (TOG), 33(4), 1-12.

[0008] However, the evaluation method disclosed in Non-Patent Document 1 is highly subjective, and annotations tend to be unstable. In addition, since the evaluation is based on which is brighter, it is not possible to evaluate hue, and it is highly likely that the method is insufficient for evaluating albedo.

[0009] Therefore, there is a need for a method that can accurately calculate the albedo of real-world image data, without the need to use the evaluation method disclosed in Non-Patent Document 1.

[0010] The present invention has been made in view of the above circumstances, and has as its object to provide an albedo calculation technique that enables the albedo of real-world image data to be calculated with high accuracy.

[0011] In order to solve the above problems, an image processing device according to one aspect of the present invention includes a specific frequency laser reflectance calculation unit, an incident light calculation unit, a measured light calculation unit, and an albedo calculation unit. The specific frequency laser reflectance calculation unit calculates a specific frequency laser reflectance, which is the reflectance at the wavelength of the specific frequency laser, from the specific frequency laser reflection intensity, which is the measurement result of the specific frequency laser on an object in the real world. The incident light calculation unit calculates incident light values, which are the values ​​of the white board data at each of multiple wavelengths that include the wavelength of the specific frequency laser, from white board data, which is the result of capturing an image of a white board with a multispectral camera. The measured light calculation unit calculates measured light values, which are the values ​​of the multispectral camera data at each of multiple wavelengths that include the wavelength of the specific frequency laser, from multispectral camera data, which is the result of capturing an image of an object with the multispectral camera. The albedo calculation unit calculates albedo, which is the reflectance specific to the object, from the specific frequency laser reflectance, the incident light value, and the measured light value.

[0012] According to one aspect of the present invention, an albedo calculation technique can be provided that enables accurate calculation of albedo by calculating a specific frequency laser reflectance from a specific frequency laser reflection intensity and using this specific frequency laser reflectance to eliminate sunlight conditions from real-world image data.

[0013] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the software configuration of the image processing device. FIG. 3 is a diagram showing an example of hyperspectral camera data received by an input information acquisition unit of the image processing device. FIG. 4 is a diagram showing an example of LiDAR reflectance intensity received by the input information acquisition unit. FIG. 5 is a diagram showing infrared wavelength data included in the hyperspectral camera data of FIG. 4. FIG. 6 is a diagram showing the albedo for the hyperspectral camera data of FIG. 4, which is the calculation result of an albedo calculation unit of the image processing device. FIG. 7 is a flowchart showing an example of the procedure and processing content of image processing performed by a control unit of the image processing device. FIG. 8 is a block diagram showing an example of the software configuration of an image processing device according to a second embodiment of the present invention. FIG. 9 is a flowchart showing an example of the procedure and processing content of image processing performed by a control unit of the image processing device. FIG. 10 is a block diagram showing an example of the software configuration of an image processing device according to a third embodiment of the present invention. FIG. 11 is a diagram showing an example of the content of a spectral information database created by a densification unit of the image processing device according to the third embodiment. FIG. 12 is a flowchart showing an example of the procedure and processing content of image processing performed by a control unit of the image processing device according to the third embodiment. Fig. 13 is a schematic diagram showing the procedure of densification calculation processing by the densification unit. Fig. 14 is a diagram showing an example of densification LiDAR reflection intensity as a result of densification by the densification unit. Fig. 15 is a block diagram showing an example of the software configuration of an image processing device according to a fourth embodiment of the present invention. Fig. 16 is a flowchart showing an example of the processing procedure and processing content of image processing executed by a control unit of the image processing device according to the fourth embodiment.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] 1 and 2 are block diagrams showing an example of the hardware and software configurations of an image processing device 1 according to a first embodiment of the present invention. The image processing device 1 may be a server computer connected to a network NW such as the Internet or a LAN (Local Area Network), or may be a user computer such as a personal computer (PC), smartphone, or tablet terminal that can be connected to the network NW.

[0016] The image processing device 1 includes a control unit 11, to which a storage unit having a program storage unit 12 and a data storage unit 13, a communication interface unit 14, and an input / output interface unit 15 are connected via a bus 16. In Fig. 1, "interface" is abbreviated as "IF."

[0017] The control unit 11 is a hardware processor such as a CPU (Central Processing Unit). For example, the CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded CPU. The control unit 11 may include multiple hardware processors.

[0018] Under the control of the control unit 11, the communication interface unit 14 transmits and receives information to and from other devices using a communication protocol defined by the network NW.

[0019] An input device 171 and an output device 172 are connected to the input / output interface unit 15. Note that the image processing device 1 does not necessarily have to be equipped with all of these devices, and some of the devices may be connected as external devices.

[0020] The input device 171 is used by a user of the image processing device 1 to input instructions and information necessary for the operation of the image processing device 1. If the image processing device 1 is a personal computer, the input device 171 may include, for example, a keyboard, a pointing device such as a mouse or a touchpad, etc. If the image processing device 1 is a smartphone or a tablet terminal, the input device 171 may include, for example, a touch panel or operation buttons arranged on the display screen of the output device 172. The input device 171 may also include a reader device for reading information from a recording medium storing various information such as programs and data. The input device 171 may also include a microphone for capturing audio, or a camera for capturing video, such as a webcam or a general video camera. The camera may also include a multispectral camera such as a hyperspectral camera. The input device 171 may also include a specific frequency laser sensor such as a LiDAR sensor for measuring distance.

[0021] The output device 172 includes a display, such as a liquid crystal monitor, an organic electroluminescence (EL) monitor, a projector screen, or a head-mounted display, that displays various information generated by the image processing device 1. The output device 172 may also include a speaker that transmits various information to the user by sound. Furthermore, the output device 172 may include a writer device that writes the information generated by the image processing device 1 to a recording medium.

[0022] The program storage unit 12 is configured by combining, for example, a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or an EEPROM (registered trademark) (electrically erasable programmable read-only memory) that can be written to and read from at any time, with a nonvolatile memory such as a read-only memory (ROM). The program storage unit 12 stores middleware such as an operating system (OS), as well as an image processing program, which is an application program required for the image processing device according to the first embodiment of the present invention to operate. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0023] The data storage unit 13 is, for example, a combination of a nonvolatile memory that can be written to and read from as needed, such as an HDD, SSD, EEPROM, or memory card, as a storage medium, and a volatile memory, such as a RAM (Random Access Memory). The data storage unit 13 includes, in its storage area, the following storage units necessary for implementing the first embodiment: an input information storage unit 131, a LiDAR wavelength reflectance storage unit 132, an incident light storage unit 133, a measurement light storage unit 134, an albedo storage unit 135, and an evaluation value storage unit 136.

[0024] The input information storage unit 131 stores input information received by the communication interface unit 14 from an external device via the network NW, or input information input from the input device 171 via the input / output interface unit 15. The input information includes, for example, data measured by a hyperspectral camera, which is an example of a multispectral camera. The data measured by the hyperspectral camera includes whiteboard data W obtained by photographing a whiteboard with the hyperspectral camera before actual photography of a real-world object, and hyperspectral camera data H, which is an example of multispectral camera data obtained by photographing a real-world object. The input information also includes LiDAR reflection intensity L, which is an example of a specific-frequency laser reflection intensity, measured by a LiDAR sensor, which is an example of a specific-frequency laser sensor. In distance measurement using a LiDAR sensor, the LiDAR reflection intensity L, which is the intensity of measurement light relative to irradiated light, is also measured. The image processing device 1 according to the first embodiment acquires this LiDAR reflection intensity L as one piece of input information. In the following description, it is assumed that the multispectral camera and multispectral camera data are the hyperspectral camera and hyperspectral camera data H, and the specific frequency laser sensor and specific frequency laser reflection intensity are the LiDAR sensor and LiDAR reflection intensity L. Note that the hyperspectral camera data H includes the wavelength of the laser used by this LiDAR within its measurement wavelength range.

[0025] The LiDAR wavelength reflectance memory unit 132 stores the reflectance at the LiDAR wavelength calculated from the LiDAR reflection intensity L by the control unit 11.

[0026] The incident light storage unit 133 stores the spectral information of the incident light calculated by the control unit 11 from the whiteboard data W. The spectral information of the incident light also includes the wavelength value of the LiDAR.

[0027] The measurement light storage unit 134 stores the spectral information of the measurement light calculated by the control unit 11 from the hyperspectral camera data H. The spectral information of the measurement light also includes the wavelength value of the LiDAR.

[0028] The albedo memory unit 135 stores the albedo calculated by the control unit 11 from the reflectance at the LiDAR wavelength, the spectral information of the incident light, and the spectral information of the measurement light.

[0029] The control unit 11 includes, as processing function units according to the first embodiment of the present invention, an input information acquisition unit 111, a LiDAR reflectance calculation unit 112, an incident light calculation unit 113, a measurement light calculation unit 114, an albedo calculation unit 115, and an output unit 116. These processing function units 111 to 116 are all realized by causing a hardware processor of the control unit 11 to execute an image processing program according to the first embodiment of the present invention, which is stored in the program storage unit 12.

[0030] The image processing program may be stored in advance in the program storage unit 12, or may be read out from a storage medium storing the image processing program when necessary and stored in the program storage unit 12. Alternatively, the image processing program may be downloaded from a program server (not shown) or the like and stored in the program storage unit 12. At least some of the processing functions of at least one of the processing function units 111 to 116 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), instead of being realized by the image processing program and a hardware processor of the control unit 11.

[0031] The input information acquisition unit 111 accepts input information sent from an external device via the network NW received by the communication interface unit 14 or input information input from the input device 171 via the input / output interface unit 15, and stores the input information in the data storage unit 13. Specifically, the input information acquisition unit 111 accepts hyperspectral camera data H, which is image data obtained by photographing a real-world object with a hyperspectral camera, and stores the data in the input information storage unit 131. Furthermore, the input information acquisition unit 111 accepts whiteboard data W, which is image data obtained by photographing a whiteboard with the hyperspectral camera, and stores the data in the input information storage unit 131. The input information acquisition unit 111 also accepts LiDAR reflection intensity L, which is the measurement result of the LiDAR sensor on the object photographed by the hyperspectral camera, and stores the data in the input information storage unit 131. In this way, the input information acquisition unit 111 is an example of an input unit that accepts specific frequency laser reflection intensity, which is the measurement result of a real-world object by a specific frequency laser sensor that uses a specific frequency laser, multispectral camera data containing the wavelength of the specific frequency laser, which is the result of photographing an object with a multispectral camera, and whiteboard data, which is the result of photographing a whiteboard with a multispectral camera.

[0032] Fig. 3 is a diagram showing an example of hyperspectral camera data H received by the input information acquisition unit 111. Fig. 3 shows an example of a road sign in front of a building wall captured as a real-world object. In reality, the hyperspectral camera data H is color image data.

[0033] Fig. 4 is a diagram showing an example of the LiDAR reflection intensity L received by the input information acquisition unit 111. Fig. 4 shows an example of measurement of a real-world object captured by hyperspectral camera data H. The LiDAR reflection intensity L is a set of reflection intensity values ​​at measurement points where reflections were detected.

[0034] The LiDAR reflectance calculation unit 112 calculates the reflectance at the LiDAR wavelength from the LiDAR reflection intensity L stored in the input information storage unit 131, and stores the calculated reflectance in the LiDAR wavelength reflectance storage unit 132. Here, the LiDAR reflection intensity L is expressed by the following equation using the aperture size D_r, air attenuation η_atm, the response function η_sys of the LiDAR sensor, the object distance R, and the angle θ between the measurement surface and the laser.

[0035]

[0036] In the above equation, the aperture diameter size D_r, air attenuation η_atm, and response function η_sys of the LiDAR sensor are known, and the object distance R and the angle θ between the measurement surface and the laser can be calculated. Therefore, the LiDAR reflectance, which is the reflectance at the LiDAR wavelength, is

[0037]

[0038] can be calculated by the following formula:

[0039]

[0040] The LiDAR reflectance calculation unit 112 calculates this LiDAR reflectance ρ_(λ_L) for each measurement point at the LiDAR reflection intensity L, and stores the calculated LiDAR reflectance in the LiDAR wavelength reflectance storage unit 132. In this way, the LiDAR reflectance calculation unit 112 is an example of a specific frequency laser reflectance calculation unit that calculates a specific frequency laser reflectance, which is the reflectance at the wavelength of a specific frequency laser, from the specific frequency laser reflection intensity, which is the measurement result of the specific frequency laser on an object in the real world.

[0041] The incident light calculation unit 113 calculates spectral information of the incident light from the whiteboard data W stored in the input information storage unit 131, and stores the calculated spectral information in the incident light storage unit 133. In measurements using a hyperspectral camera, it is common to photograph the whiteboard in advance to obtain the spectrum of the incident light. Therefore, also in the image processing device 1 according to the first embodiment, the incident light calculation unit 113 calculates the value of the incident light at an arbitrary wavelength λ from the whiteboard data W.

[0042]

[0043] and the wavelength of LiDAR

[0044]

[0045] The value of incident light at

[0046]

[0047] The arbitrary wavelength λ is one of the wavelengths within the wavelength band of the hyperspectral camera data H. The incident light calculation unit 113 stores the calculated incident light value e_λ at the arbitrary wavelength λ of the whiteboard data W and the incident light value e_(λ_L) at the LiDAR wavelength λ_L in the incident light storage unit 133. Note that the incident light calculation unit 113 may store these incident light values ​​e_λ and e_(λ_L) in the incident light storage unit 133 in the form of an incident light value ratio, e_(λ_L) / e_λ. The incident light calculation unit 113 calculates multiple incident light values ​​while updating the arbitrary wavelength λ. In other words, the incident light calculation unit 113 calculates the incident light value at each of multiple wavelengths including the LiDAR wavelength within the wavelength band of the hyperspectral camera data H. Based on the LiDAR reflection intensity L, the incident light calculation unit 113 calculates the incident light value at each of a plurality of wavelengths containing the LiDAR wavelength for each measurement point of the white board data W corresponding to each measurement point in the LiDAR reflection intensity L. In this way, the incident light calculation unit 113 is an example of an incident light calculation unit that calculates, from the white board data that is the result of capturing an image of the white board with a multispectral camera, incident light values ​​that are values ​​of the white board data at each of a plurality of wavelengths that contain the wavelength of the specific frequency laser.

[0048] The measurement light calculation unit 114 calculates spectral information of the measurement light from the hyperspectral camera data H stored in the input information storage unit 131, and stores the calculated spectral information in the measurement light storage unit 134. Specifically, the measurement light calculation unit 114 calculates a measurement value at an arbitrary wavelength λ in the hyperspectral camera data H.

[0049]

[0050] and the measurement value at the LiDAR wavelength in the hyperspectral camera data H

[0051]

[0052] and are calculated. The arbitrary wavelength λ is the same as the arbitrary wavelength λ calculated by the incident light calculation unit 113. The measurement light calculation unit 114 stores the measurement value H_λ at the arbitrary wavelength λ and the measurement value H_(λ_L) at the LiDAR wavelength in the calculated hyperspectral camera data H in the measurement light storage unit 134. Figure 5 is a diagram showing infrared wavelength data included in the hyperspectral camera data H of Figure 4 when the arbitrary wavelength λ is an infrared wavelength. Note that the measurement light calculation unit 114 may store these measurement values ​​H_λ and H_(λ_L) in the measurement light storage unit 134 in the form of a measurement light ratio, H_λ / H_(λ_L). The measurement light calculation unit 114 calculates multiple measurement values ​​while updating the arbitrary wavelength λ. In other words, the measurement light calculation unit 114 calculates measurement values ​​at each of multiple wavelengths including LiDAR wavelengths within the wavelength band of the hyperspectral camera data H. Based on the LiDAR reflection intensity L, the measurement light calculation unit 114 calculates a measurement value at each of a plurality of wavelengths containing the LiDAR wavelength for each measurement point of the hyperspectral camera data H corresponding to each measurement point in the LiDAR reflection intensity L. In this way, the measurement light calculation unit 114 is an example of a measurement light calculation unit that calculates a measurement value, which is the value of the multispectral camera data at each of a plurality of wavelengths containing the wavelength of the specific frequency laser, from the multispectral camera data that is the result of capturing an object with a multispectral camera.

[0053] The albedo calculation unit 115 calculates the albedo A based on the LiDAR reflectance ρ(λ_L) stored in the LiDAR wavelength reflectance memory unit 132, the incident light e_λ and e_(λ_L) stored in the incident light memory unit 133, and the measurement values ​​H_λ and H_(λ_L) stored in the measurement light memory unit 134, and stores the calculated albedo A in the albedo memory unit 135.

[0054] Generally, the captured data measures specular and diffuse reflection on the object surface. Since the intrinsic image decomposition task assumes a perfectly diffuse reflecting Lambertian surface, the measured value H_λ of an arbitrary wavelength λ in the hyperspectral camera data H is expressed as follows:

[0055]

[0056] Here, m is a function that represents the shape of the object. From this, the measured value H_λ at an arbitrary wavelength λ can be approximated as follows:

[0057]

[0058] Here, if we calculate the ratio between an arbitrary wavelength λ and the LiDAR wavelength λ_L, we get the following:

[0059]

[0060] This formula can be transformed into a formula for calculating the measurement light reflectance ρ_λ, which is the reflectance at an arbitrary wavelength λ, as follows:

[0061]

[0062] Here, e_(λ_L) / e_λ is known from measurement of the white board, and H_λ / H_(λ_L) is known from measurement with the hyperspectral camera. The LiDAR reflectance ρ_(λ_L) is also calculated from the LiDAR reflection intensity. Therefore, the measured light reflectance ρ_λ at any wavelength λ can be calculated. The albedo calculation unit 115 calculates the measured light reflectance at each of multiple wavelengths within the wavelength band of the hyperspectral camera data H. For each measurement point of the hyperspectral camera data H corresponding to each measurement point in the LiDAR reflection intensity L, the albedo calculation unit 115 calculates the measured light reflectance at each of multiple wavelengths containing the LiDAR wavelength.

[0063] Thereafter, the albedo calculation unit 115 calculates the albedo A by converting the calculated multiple measured light reflectances into the RGB color space using the CIE 1931 model or the like. The albedo calculation unit 115 then stores the calculated albedo A in the albedo storage unit 135. Figure 6 is a diagram showing the albedo for the hyperspectral camera data of Figure 4, which is the calculation result of the albedo calculation unit 115. In this way, the albedo calculation unit 115 is an example of an albedo calculation unit that calculates the albedo, which is the reflectance specific to an object, from the specific frequency laser reflectance, the incident light value, and the measurement value.

[0064] The output unit 116 transmits the albedo A stored in the albedo storage unit 135 to another device via the network NW using the communication interface unit 14, or outputs it to the output device 172 via the input / output interface unit 15. For example, if the output device 172 is a display, an image representing the albedo A is displayed. Furthermore, if the output device 172 is a writer device, the albedo A is written as a file to a recording medium.

[0065] (Operation Example) Next, an operation example of the image processing device 1 configured as described above will be described. Fig. 7 is a flowchart showing an example of the processing procedure and processing content of image processing executed by the control unit 11 in accordance with the image processing program stored in the program storage unit 12 of the image processing device 1. The image processing shown in this flowchart is executed in response to a processing start instruction sent via the network NW received by the communication interface unit 14 or input from the input device 171 via the input / output interface unit 15.

[0066] As shown in Figure 7, the control unit 11 of the image processing device 1 first operates as an input information acquisition unit 111 to acquire input information transmitted from another device of any source via the network NW or information input from the input device 171 via the input / output interface unit 15, and stores the information in the input information storage unit 131 (step S101).

[0067] Then, the control unit 11 operates as the LiDAR reflectance calculation unit 112 and calculates the LiDAR reflectance ρ_(λ_L) for the measurement point to be processed, which is one measurement point, from the LiDAR reflection intensity L stored in the input information storage unit 131 (step S102). The control unit 11 stores this calculated LiDAR reflectance ρ_(λ_L) in the LiDAR wavelength reflectance storage unit 132.

[0068] The control unit 11 also operates as an incident light calculation unit 113 and calculates, as incident light spectral information, incident light e_λ at an arbitrary wavelength λ in the hyperspectral camera data H and incident light e_(λ_L) at the LiDAR wavelength, from the value corresponding to the measurement point to be processed in the LiDAR reflection intensity L of the whiteboard data W stored in the input information storage unit 131 (step S103). The control unit 11 stores the calculated incident light e_λ and e_(λ_L) in the incident light storage unit 133. Note that the control unit 11 may also calculate the incident light ratio e_(λ_L) / e_λ and store it in the incident light storage unit 133.

[0069] Next, the control unit 11 operates as the measurement light calculation unit 114 and calculates, as spectral information of the measurement light, a measurement value H_λ at an arbitrary wavelength λ in the hyperspectral camera data H and a measurement value H_(λ_L) at the LiDAR wavelength from the value of the measurement point of the hyperspectral camera data H corresponding to the measurement point to be processed at the LiDAR reflection intensity L stored in the input information storage unit 131 (step S104). The control unit 11 stores the calculated measurement values ​​H_λ and H_(λ_L) in the measurement light storage unit 134. Note that the control unit 11 may also calculate a measurement light ratio, H_λ / H_(λ_L), and store the ratio in the measurement light storage unit 134.

[0070] The control unit 11 then operates as the albedo calculation unit 115 to calculate the albedo A. Specifically, the control unit 11 first calculates the measurement light reflectance ρ_λ, which is the reflectance at an arbitrary wavelength λ at the measurement point of the hyperspectral camera data H corresponding to the measurement point to be processed at the LiDAR reflection intensity L, based on the LiDAR reflectance ρ_(λ_L) stored in the LiDAR wavelength reflectance storage unit 132, the incident light e_λ and e_(λ_L) stored in the incident light storage unit 133, and the measurement values ​​H_λ and H_(λ_L) stored in the measurement light storage unit 134 (step S105). The control unit 11 temporarily stores the calculated measurement light reflectance ρ_λ in the albedo storage unit 135.

[0071] Thereafter, the control unit 11 determines whether the measured light reflectance has been calculated for all wavelengths in the wavelength band of the hyperspectral camera data H (step S106). Note that the wavelengths to be measured may be multiple wavelengths skipped by a predetermined number of wavelengths, or multiple wavelengths may be specified in advance. The control unit 11 may also receive multiple wavelengths to be measured specified by a user or an external device via the communication interface unit 14 or the input / output interface unit 15. If the measured light reflectance has not yet been calculated for all wavelengths, the control unit 11 proceeds to the processing of step S103 and repeats the processing of steps S103 to S106 for the next wavelength.

[0072] If it is determined in step S106 that the measured light reflectances have been calculated for all wavelengths, the control unit 11 calculates the albedo A by converting the measured light reflectances temporarily stored in the albedo storage unit 135 into the RGB color space using the CIE 1931 model or the like (step S107). The control unit 11 stores the calculated albedo A in the albedo storage unit 135.

[0073] Thereafter, the control unit 11 determines whether or not the albedo has been calculated for all measurement points at the LiDAR reflection intensity L (step S108). If there are measurement points for which the albedo has not yet been calculated, the control unit 11 proceeds to the processing of step S102, and repeats the processing of steps S102 to S108 for the next measurement point as the measurement point to be processed.

[0074] If it is determined in step S108 that the albedo has been calculated for all measurement points, the control unit 11 operates as the output unit 116 and outputs the albedo stored in the albedo storage unit 135 (step S109).The control unit 11 then ends the processing shown in this flowchart.

[0075] (Actions and Effects) As described above, the image processing device 1 according to the first embodiment calculates, by the LiDAR reflectance calculation unit 112, the LiDAR reflectance ρ_(λ_L), which is the reflectance at the LiDAR wavelength, from the LiDAR reflection intensity L, which is the measurement result of the LiDAR sensor for an object in the real world, and also calculates, by the incident light calculation unit 113, incident light values, which are the values ​​of the whiteboard data W at each of a plurality of wavelengths that include the LiDAR wavelength, from the whiteboard data W, which is the result of photographing a whiteboard with a hyperspectral camera, that is, the incident light value e_λ at any wavelength λ of the whiteboard data W and the incident light value e_(λ_L) at each LiDAR wavelength λ_L. Furthermore, the image processing device 1 calculates, via the measurement light calculation unit 114, measurement values ​​that are values ​​of the hyperspectral camera data H at each of multiple wavelengths containing the LiDAR wavelength, from the hyperspectral camera data H, which is the result of capturing an object with the hyperspectral camera. That is, the measurement value H_λ at any wavelength λ in the hyperspectral camera data H and the measurement value H_(λ_L) at the LiDAR wavelength. The image processing device 1 then calculates, via the albedo calculation unit, an albedo A, which is a reflectance specific to the object, from the calculated LiDAR reflectance, incident light value, and measurement value. According to the image processing device 1 of the first embodiment, the LiDAR reflectance is calculated from the LiDAR reflection intensity L, and the albedo A can be accurately calculated by using this LiDAR reflectance to eliminate sunlight conditions from the hyperspectral camera data H, which is real-world image data.

[0076] Second Embodiment The same reference numerals as those in the first embodiment are used to designate the same configurations and processes as those in the first embodiment, and descriptions thereof will be omitted, and only the portions that differ from the first embodiment will be described.

[0077] (Configuration Example) The hardware configuration of the image processing device 1 according to the second embodiment is similar to that of the first embodiment.

[0078] 8 is a block diagram showing an example of the software configuration of an image processing device 1 according to a second embodiment of the present invention. In the image processing device 1 according to the second embodiment, the control unit 11 further includes an evaluation unit 117 as a processing function unit according to the second embodiment. The evaluation unit 117 performs quantitative evaluation of the albedo A stored in the albedo storage unit 135. A specific method of this quantitative evaluation will be described later. Furthermore, in the image processing device 1 according to the second embodiment, the data storage unit 13 further includes an evaluation value storage unit 136 in its storage area as a storage unit necessary for implementing the second embodiment. The evaluation value storage unit 136 stores the quantitative evaluation results obtained by the evaluation unit 117.

[0079] As will be described later, the operation of the input information acquisition unit 111 and the output unit 116 of the control unit 11 in the second embodiment is different from that in the first embodiment. Furthermore, the stored contents of the input information storage unit 131 of the data storage unit 13 in the second embodiment are different from those in the first embodiment.

[0080] (Operation Example) Next, an operation example of the image processing device 1 according to the second embodiment configured as described above will be described. Fig. 9 is a flowchart showing an example of the processing procedure and processing content of image processing executed by the control unit 11 in accordance with the image processing program stored in the program storage unit 12 of the image processing device 1 according to the second embodiment. The image processing shown in this flowchart is executed in response to a processing start instruction sent via the network NW received by the communication interface unit 14 or input from the input device 171 via the input / output interface unit 15.

[0081] 9, the control unit 11 of the image processing device 1 first operates as an input information acquisition unit 111 in the process of step S101, as described in the first embodiment, to acquire input information transmitted from another device of any transmission source via the network NW or information input from the input device 171 via the input / output interface unit 15, and stores the information in the input information storage unit 131. However, in this second embodiment, the input information includes the albedo obtained by intrinsic image decomposition in addition to the whiteboard data W, hyperspectral camera data H, and LiDAR reflection intensity L of the first embodiment.

[0082]

[0083] This albedo A_^ obtained by eigenimage decomposition is a result of an external device executing a known eigenimage decomposition task on the hyperspectral camera data H, which is one piece of input information. In this way, the control unit 11 operating as the input information acquisition unit 111 is an example of an input unit that accepts the specific frequency laser reflection intensity, which is the measurement result of a real-world object by a specific frequency laser sensor that uses a specific frequency laser, multispectral camera data containing the wavelength of the specific frequency laser, which is the result of capturing an object with a multispectral camera, whiteboard data, which is the result of capturing an image of a whiteboard with the multispectral camera, and the albedo obtained by the eigenimage decomposition task by the external device.

[0084] Thereafter, the control unit 11 performs the same processes as in the first embodiment, from step S102 to step S108, thereby calculating the albedo A, and the calculated albedo A is stored in the albedo storage unit 135.

[0085] The albedo A obtained by the above calculation is sparse (low density). Therefore, in the second embodiment, the control unit 11 operates as the evaluation unit 117 and uses the value of this albedo A to quantitatively evaluate the albedo A^ estimated by the eigenimage decomposition model (step S111). Specifically, the control unit 11 uses a mask M of an area where the value of albedo A stored in the albedo storage unit 135 exists to quantitatively evaluate the albedo A^ estimated by the eigenimage decomposition model stored in the input information storage unit 131. That is, the quantitative evaluation E is calculated as follows:

[0086]

[0087] Here, D(a, b) is a function representing the distance between a and b, such as the mean squared error (MSE). In this quantitative evaluation, the albedo A^ estimated by the eigenimage decomposition model can be evaluated using the absolute albedo A value calculated by the albedo calculation unit, rather than the relative reflection intensity as disclosed in Non-Patent Document 1. It is also possible to quantitatively evaluate the hue. The control unit 11 stores the results of the quantitative evaluation E in the evaluation value storage unit 136. In this way, the control unit 11 operating as the evaluation unit 117 is an example of an evaluation unit that quantitatively evaluates the albedo estimated by the eigenimage decomposition task from multispectral camera data, which is the result of capturing an object with a multispectral camera, using the albedo calculated by the albedo calculation unit.

[0088] Then, the control unit 11 operates as the output unit 116, and in the second embodiment, outputs the result of the quantitative evaluation E stored in the evaluation value storage unit 136, rather than the albedo (step S112). Then, the control unit 11 ends the processing shown in this flowchart.

[0089] (Actions and Effects) As described above, the image processing device 1 according to the second embodiment quantitatively evaluates the albedo A^ estimated by an externally input eigenimage decomposition task using the albedo A calculated by the albedo calculation unit 115, using the evaluation unit 117. In addition to the effects of the first embodiment, the image processing device 1 according to the second embodiment can quantitatively evaluate the albedo A^ estimated by the eigenimage decomposition model using the absolute albedo A value, rather than the relative reflection intensity. Furthermore, it becomes possible to quantitatively evaluate the albedo A^ in terms of hue as well.

[0090] [Third Embodiment] The same reference numerals as those in the first and second embodiments are used to denote configurations and processes similar to those in the second embodiment, and descriptions thereof will be omitted, and only the parts that differ from the first and second embodiments will be described.

[0091] (Configuration Example) The hardware configuration of the image processing device 1 according to the third embodiment is similar to that of the first embodiment.

[0092] FIG. 10 is a block diagram showing an example of the software configuration of an image processing device 1 according to a third embodiment of the present invention. In the image processing device 1 according to the third embodiment, the control unit 11 further includes a densification unit 118 as a processing function unit according to the third embodiment, in addition to the processing function unit according to the second embodiment. The densification unit 118 densifies the LiDAR reflection intensity L. The densification unit 118 is an example of a densification unit that acquires a densified specific frequency laser reflection intensity by densifying the specific frequency laser reflection intensity using multispectral camera data. In addition, in the image processing device 1 according to the third embodiment, the data storage unit 13 further includes, in its storage area, a spectral information database 137 and a densified LiDAR reflection intensity storage unit 138 as storage units necessary for implementing the third embodiment. The spectral information database 137 is a dictionary used by the densification unit 118 when densifying the LiDAR reflection intensity L, and its contents are registered by the densification unit 118. The densified LiDAR reflection intensity memory unit 138 stores the densified LiDAR reflection intensity L′ densified by the densification unit 118.

[0093] Furthermore, in the LiDAR reflectance calculation unit 112, incident light calculation unit 113, and measurement light calculation unit 114 of the control unit 11 in the third embodiment, the measurement point to be processed is used as the measurement point in the high-density LiDAR reflection intensity L′ stored in the high-density LiDAR reflection intensity storage unit 138. Furthermore, the evaluation method of the evaluation unit 117 is also different from that of the second embodiment, as will be described later.

[0094] Here, the densification unit 118 will be described in detail. The densification unit 118 executes two main processes. One is a densification calculation process that performs densification calculations, and the other is a dictionary creation process that creates a dictionary to be used in the densification calculation process prior to the densification calculation process.

[0095] First, the dictionary creation process will be described. The densification unit 118 creates a database representing the correspondence between the hyperspectral camera data H stored in the input information storage unit 131 and the LiDAR reflection intensity L stored in the LiDAR wavelength reflectance storage unit 132, as the spectral information database 137. Fig. 11 is a diagram showing an example of the contents of the spectral information database 137 created by the densification unit 118. The spectral information database 137 is a database in which the values ​​of the LiDAR reflection intensity L and the values ​​of the hyperspectral camera data H are registered in association with pixel numbers. The pixel numbers are identification numbers of records in the database.

[0096] The densification unit 118 acquires pixel values ​​of the hyperspectral camera data H corresponding to regions where values ​​are entered in the LiDAR reflection intensity L, and checks whether there is a record in the spectral information database 137 where the two values ​​match. The value of each pixel of the hyperspectral camera data H is spectral information made up of multiple wavelength values. The multiple wavelengths include LiDAR wavelengths. If there is no matching record, the densification unit 118 issues a new pixel number and adds a record having the acquired value of the LiDAR reflection intensity L and the value of the hyperspectral camera data H to the spectral information database 137. In this way, the densification unit 118 is an example of a dictionary creation unit that creates a database showing the correspondence between specific frequency laser reflection intensity and multispectral camera data from the input specific frequency laser reflection intensity and multispectral camera data.

[0097] Next, the densification calculation process will be described. The densification unit 118 performs a densification calculation of the LiDAR reflection intensity L based on the spectral information database 137 to generate a densified LiDAR reflection intensity L′, and stores the generated densified LiDAR reflection intensity L′ in the densified LiDAR reflection intensity storage unit 138. Specifically, the densification unit 118 uses the spectral information database 137 to set the LiDAR reflection intensity of the pixel number most similar to the spectrum of the pixel of the hyperspectral camera data H corresponding to the complement target region for which a value is not entered in the LiDAR reflection intensity L stored in the LiDAR wavelength reflectance storage unit 132 as the value of the complement target region. A specific calculation method used by the densification unit 118 will be described later. In this way, the densification unit 118 is an example of a densification unit that performs a densification calculation of a specific frequency laser reflection intensity based on the created database to generate a densified specific frequency laser reflection intensity.

[0098] (Operation Example) Next, an operation example of the image processing device 1 configured as described above will be described. Fig. 12 is a flowchart showing an example of the processing procedure and processing content of image processing executed by the control unit 11 in accordance with the image processing program stored in the program storage unit 12 of the image processing device 1. The image processing shown in this flowchart is executed in response to a processing start instruction sent via the network NW received by the communication interface unit 14 or input from the input device 171 via the input / output interface unit 15.

[0099] As shown in Figure 5, the control unit 11 of the image processing device 1 first operates as an input information acquisition unit 111 in the processing of step S101, as in the second embodiment, to acquire input information and store it in the input information storage unit 131.

[0100] Thereafter, the control unit 11 operates as the densification unit 118 to create a database serving as a dictionary from the hyperspectral camera data H and the LiDAR reflection intensity L as the spectral information database 137 (step S121). Generally, the LiDAR reflection intensity L is sparse (low density) compared to the hyperspectral camera data H. Therefore, in this embodiment, first, when a set T of pixels having a LiDAR reflection intensity value and a set F of pixels having no value are defined,

[0101]

[0102] For pixels that satisfy the above condition, the value L_t of the LiDAR reflection intensity L and the value H_t of the hyperspectral camera data H are associated with each other to create a spectral information database 137 in the data storage unit 13.

[0103] Thereafter, the control unit 11 calculates the densified LiDAR reflection intensity L′ (step S122. That is, the control unit 11 sets a pixel L_f that has no value in the LiDAR reflection intensity L stored in the input information storage unit 131 as a pixel to be complemented, and calculates an optimal value for this pixel L_f to be complemented.

[0104] FIG. 13 is a schematic diagram showing the procedure for the densification calculation process for this calculation. The value H_f of the hyperspectral camera data H represents the characteristics of the object surface. Areas in this hyperspectral camera data H with high similarity in spectral information are likely to have high similarity in the object surface characteristics, and as a result, similar values ​​of the LiDAR reflection intensity L. Therefore, the control unit 11 reads spectral information, which is the value of the pixel in the hyperspectral camera data H corresponding to the pixel L_f to be complemented in the spectral information database 137, from the input information storage unit 131. In the example of FIG. 13 , the spectral information (0, 10, 12, ..., 103) is read from the input information storage unit 131.

[0105] Then, the control unit 11 refers to the spectrum information database 137 using this spectrum information, and calculates the pixel t' having the most similar spectrum information using the following formula.

[0106]

[0107] Note that D(a, b) may be a function representing distance as described in the second embodiment, or may be a function representing similarity. For example, cosine similarity is used as a function representing similarity. In the example of FIG. 13 , the spectral information of pixel number 1 is most similar to the spectral information of (0, 10, 12, ..., 103). Therefore, pixel number 1 is acquired as pixel t' having the most similar spectral information.

[0108] Therefore, the control unit 11 determines to use the value of this pixel number L_t′ in the spectrum information database 137 as the LiDAR reflection intensity L of the pixel L_f to be complemented. In the example of Fig. 13 , the LiDAR reflection intensity L of the pixel L_f to be complemented is "100", which is the LiDAR reflection intensity L of pixel number 1.

[0109] The control unit 11 stores the LiDAR reflection intensity L read from the LiDAR wavelength reflectance storage unit 132 in the high-density LiDAR reflection intensity storage unit 138. Then, the value of a pixel that has no value in the stored LiDAR reflection intensity L is replaced with the value of the determined pixel number L_t′.

[0110] The control unit 11 performs such densification calculation processing as follows:

[0111]

[0112] This is repeated for all t to complement all defects in the input LiDAR reflection intensity L. As a result, the high-density LiDAR reflection intensity storage unit 138 stores a high-density LiDAR reflection intensity L' in which all defects have been complemented. Figure 14 is a diagram showing an example of this high-density LiDAR reflection intensity L'.

[0113] Furthermore, when using hyperspectral camera data H, an imbalance in shade or illumination may occur. In such cases, not only are the spectra similar, but values ​​located closer to a pixel L_f that does not have a value for LiDAR reflection intensity L may be more reliable. In such cases, position information may be embedded as follows:

[0114]

[0115] Here, w is a constant that represents the weight and is changed depending on the application.

[0116] Once the high-density LiDAR reflection intensity L' has been calculated in this way, the control unit 11 operates as the LiDAR reflectance calculation unit 112 in the processing of step S102 to calculate the LiDAR field of view. However, in the third embodiment, the control unit 11 calculates the LiDAR reflectance λ_L, which is the reflectance at the LiDAR wavelength λ_L, for the processing target measurement point, which is one measurement point, from the high-density LiDAR reflection intensity L' stored in the high-density LiDAR reflection intensity storage unit 138.

[0117]

[0118] The LiDAR reflectance ρ_(λ_L′) can be calculated using an equation similar to equation 3. The control unit 11 stores the calculated LiDAR reflectance ρ_(λ_L′) in the LiDAR wavelength reflectance storage unit 132.

[0119] Furthermore, in the processing of step S103, the control unit 11 operates as the incident light calculation unit 113 to calculate the incident light. However, in the third embodiment, the control unit 11 calculates the incident light spectral information from the value corresponding to the measurement point to be processed in the densified LiDAR reflection intensity L' of the whiteboard data W stored in the input information storage unit 131, and calculates the incident light e_λ at an arbitrary wavelength λ in the hyperspectral camera data H and the incident light value at the LiDAR wavelength.

[0120]

[0121] The control unit 11 stores the calculated incident light e_λ and e_(λ_L') in the incident light storage unit 133. Note that the control unit 11 may also calculate the incident light ratio e_(λ_L') / e_λ and store it in the incident light storage unit 133.

[0122] Next, in the process of step S104, the control unit 11 operates as the measurement light calculation unit 114 to calculate the measurement light. However, in the third embodiment, the control unit 11 calculates, as spectral information of the measurement light, a measurement value H_λ at an arbitrary wavelength λ in the hyperspectral camera data H and a measurement value H_λ at a high-density LiDAR wavelength from the value of the measurement point of the hyperspectral camera data H corresponding to the measurement point to be processed in the LiDAR reflection intensity L stored in the input information storage unit 131.

[0123]

[0124] The control unit 11 stores the calculated measurement values ​​H_λ and H_(λ_L') in the measurement light storage unit 134. Note that the control unit 11 may calculate a measurement light ratio, H_λ / H_(λ_L'), and store the ratio in the measurement light storage unit 134.

[0125] Thereafter, the control unit 11 performs the processes of steps S105 to S108 as described in the first embodiment.

[0126] By the above processing, in this third embodiment, the albedo is calculated using the densified LiDAR reflection intensity, thereby obtaining a high-density albedo A' rather than the sparse (low-density) albedo as in the first and second embodiments.

[0127] Then, in the process of step S111, the control unit 11 operates as the evaluation unit 117 and performs quantitative evaluation of the albedo A^ estimated by the eigenimage decomposition model. In the third embodiment, the high-density albedo A' value is used for this quantitative evaluation. That is, the control unit 11 uses the high-density albedo A' stored in the albedo storage unit 135 to quantitatively evaluate the albedo A^ estimated by the eigenimage decomposition model stored in the input information storage unit 131. Specifically, the control unit 11 calculates the quantitative evaluation E as follows:

[0128]

[0129] Then, in the process of step S112, the control unit 11 operates as the output unit 116 to output the results of the quantitative evaluation E stored in the evaluation value storage unit 136, and then ends the process shown in this flowchart.

[0130] (Actions and Effects) As described above, in the image processing device 1 according to the third embodiment, the densification unit 118 acquires a densified LiDAR reflection intensity L' by densifying the LiDAR reflection intensity L using the hyperspectral camera data H. According to the image processing device 1 according to the third embodiment, it is possible to obtain a high-density albedo A' by using the densified LiDAR reflection intensity L'.

[0131] In the image processing device 1 according to the third embodiment, the densification unit 118 creates a spectral information database 137 representing the correspondence between the LiDAR reflection intensity L and the hyperspectral camera data H from the input LiDAR reflection intensity L and the hyperspectral camera data H, and performs a densification calculation of the LiDAR reflection intensity L based on the created spectral information database 137 to generate a densified specific frequency laser reflection intensity L'. According to the image processing device 1 according to the third embodiment, by using the hyperspectral camera data H, it is possible to accurately densify the LiDAR reflection intensity L even for colorful objects whose reflection characteristics change significantly depending on the wavelength.

[0132] In the image processing device 1 according to the third embodiment, the densification unit 118 creates the spectral information database 137 using the areas containing the values ​​of the LiDAR reflection intensity L and the corresponding values ​​of the hyperspectral camera data H. Therefore, according to the image processing device 1 according to the third embodiment, it is possible to create the spectral information database 137, which serves as a dictionary for acquiring the values ​​of the LiDAR reflection intensity L, using the values ​​of the hyperspectral camera data H as a key.

[0133] Furthermore, in the image processing device 1 according to the third embodiment, the densification unit 118 generates a densified LiDAR reflection intensity L′ by using the spectral similarity and the spectral information database 137 to calculate values ​​for areas where no value is entered in the input LiDAR reflection intensity L. Therefore, according to the image processing device 1 according to the third embodiment, it is possible to calculate values ​​for areas where no value is entered in the LiDAR reflection intensity L based on the similarity between the values ​​of the hyperspectral camera data H corresponding to areas where no value is entered in the input LiDAR reflection intensity L and the values ​​of the hyperspectral camera data H stored in the spectral information database 137.

[0134] [Fourth embodiment] The same reference symbols as those in the first and second embodiments are used for the configurations and processes similar to those in the first embodiment, and the description thereof will be omitted, and only the parts that differ from the first and second embodiments will be described.

[0135] (Configuration Example) The hardware configuration of the image processing device 1 according to the fourth embodiment is similar to that of the first embodiment.

[0136] 15 is a block diagram showing an example of the software configuration of an image processing device 1 according to a fourth embodiment of the present invention. In the image processing device 1 according to the fourth embodiment, the control unit 11 further includes an eigenimage decomposition execution unit 119 as a processing function unit according to the fourth embodiment, in addition to the processing function unit of the second embodiment. The eigenimage decomposition execution unit 119 estimates an albedo A^ obtained by eigenimage decomposition by performing a known eigenimage decomposition process on the hyperspectral camera data H stored in the input information storage unit 131. The eigenimage decomposition execution unit 119 stores the estimated albedo A^ obtained by eigenimage decomposition in the albedo storage unit 135.

[0137] In addition, in the evaluation unit 117 of the control unit 11 in the fourth embodiment, the albedo A^ obtained by the eigenimage decomposition to be evaluated is the albedo stored in the albedo storage unit 135.

[0138] (Operation Example) Next, an operation example of the image processing device 1 according to the fourth embodiment configured as described above will be described. Fig. 16 is a flowchart showing an example of the processing procedure and processing content of image processing executed by the control unit 11 in accordance with the image processing program stored in the program storage unit 12 of the image processing device 1 according to the fourth embodiment. The image processing shown in this flowchart is executed in response to a processing start instruction sent via the network NW received by the communication interface unit 14 or input from the input device 171 via the input / output interface unit 15.

[0139] As shown in Figure 16, the control unit 11 of the image processing device 1 first operates as an input information acquisition unit 111 in the processing of step S101, as described in the first embodiment, to acquire input information and store it in the input information storage unit 131.

[0140] Thereafter, in the fourth embodiment, the control unit 11 operates as the eigenimage decomposition execution unit 119, and performs a known eigenimage decomposition process on the hyperspectral camera data H stored in the input information storage unit 131 to estimate the albedo A^ by the eigenimage decomposition (step S131). The control unit 11 stores the estimated albedo A^ by the eigenimage decomposition in the albedo storage unit 135.

[0141] The subsequent processes of steps S102 to S108, S111, and S112 are as described in the second embodiment. However, the albedo A^ obtained by intrinsic image decomposition evaluated in the process of step S111 is not input from the outside and stored in the input information storage unit 131 as in the second embodiment, but is estimated in the image processing device 1 and stored in the albedo storage unit 135.

[0142] (Actions and Effects) As described above, in the image processing device 1 to which the image processing device according to the fourth embodiment is applied, as in the second embodiment, it is possible to quantitatively evaluate the albedo A^ estimated by the intrinsic image decomposition model not in terms of relative reflection intensity but in terms of the absolute value of albedo A. Furthermore, it is also possible to quantitatively evaluate the albedo A^ in terms of hue.

[0143] Furthermore, according to the image processing device 1 according to the fourth embodiment, the albedo A^ is estimated using the eigenimage decomposition model within the image processing device 1, so there is no need to input the albedo A^ from outside. Conversely, the image processing device 1 according to the fourth embodiment can be configured by incorporating the image processing device 1 according to the second embodiment into a device that estimates the albedo A^ using the eigenimage decomposition model.

[0144] [Other Embodiments] The present invention is not limited to the above-described embodiments.

[0145] For example, the image processing device 1 according to the first to fourth embodiments can utilize multispectral camera data other than hyperspectral camera data H obtained using a multispectral camera other than a hyperspectral camera, and specific frequency laser reflection intensity other than LiDAR reflection intensity L obtained using a specific frequency laser sensor other than a LiDAR sensor.

[0146] Furthermore, the method for increasing the density of the LiDAR reflection intensity L described in the image processing device 1 according to the third embodiment is just one example, and other methods may be used for increasing the density.

[0147] Furthermore, in the image processing device 1 according to the third embodiment, evaluation is performed and the evaluation result is output, but it is also possible to output a high-density albedo A' as in the first embodiment.

[0148] Moreover, the image processing device 1 according to the fourth embodiment may also incorporate the high density described in the third embodiment.

[0149] Furthermore, the flow of each process described with reference to the flowchart is not limited to the procedure described. For example, the order of the processes in step S103 and step S104 shown in FIG. 7 may be reversed, or they may be performed simultaneously in parallel. In this way, the order of some steps may be reversed, or some steps may be performed simultaneously in parallel. Furthermore, the processing content of some steps may be modified.

[0150] Furthermore, the image processing program stored in the program storage unit 12 may be transferred in a state where it is stored in the computer constituting the image processing device 1, or may be transferred in a state where it is not stored in the computer. In the latter case, the image processing program may be transferred via the network NW, or may be transferred in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium capable of storing a program and is readable by a computer, such as a CD-ROM or a memory card.

[0151] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0152] In short, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0153] DESCRIPTION OF SYMBOLS 1...Image processing device 11...Control unit 12...Program storage unit 13...Data storage unit 14...Communication interface unit 15...Input / output interface unit 16...Bus 111...Input information acquisition unit 112...LiDAR reflectance calculation unit 113...Incidence light calculation unit 114...Measurement light calculation unit 115...Albedo calculation unit 116...Output unit 117...Evaluation unit 118...Densification unit 119...Eigenimage decomposition execution unit 131...Input information storage unit 132...LiDAR wavelength reflectance storage unit 133...Incidence light storage unit 134...Measurement light storage unit 135...Albedo storage unit 136...Evaluation value storage unit 137...Spectral information database 138...Densified LiDAR reflection intensity storage unit 171...Input device 172...Output device NW...Network

Claims

1. An image processing device comprising: a specific frequency laser reflectivity calculation unit that calculates a specific frequency laser reflectivity, which is the reflectivity at the wavelength of a specific frequency laser, from a specific frequency laser reflection intensity, which is the measurement result of a specific frequency laser on an object in the real world; an incident light calculation unit that calculates an incident light value, which is the value of the white board data at each of a plurality of wavelengths that includes the wavelength of the specific frequency laser, from white board data, which is the result of photographing a white board with a multispectral camera; a measured light calculation unit that calculates a measurement value, which is the value of the multispectral camera data at each of the plurality of wavelengths that includes the wavelength of the specific frequency laser, from multispectral camera data, which is the result of photographing the object with the multispectral camera; and an albedo calculation unit that calculates an albedo, which is the reflectivity specific to the object, from the specific frequency laser reflectivity, the incident light value, and the measurement value.

2. The image processing device according to claim 1, further comprising a densification unit that densifies the specific frequency laser reflection intensity using the multispectral camera data, thereby obtaining a densified specific frequency laser reflection intensity.

3. The image processing device according to claim 1 or 2, further comprising an evaluation unit that quantitatively evaluates the albedo estimated by an eigenimage decomposition task from multispectral camera data that is the result of photographing the object with the multispectral camera, using the albedo calculated by the albedo calculation unit.

4. An image processing program that causes a processor to execute the following steps: a process of calculating a specific frequency laser reflectance, which is the reflectance at the wavelength of a specific frequency laser, from specific frequency laser reflection intensity, which is the measurement result of a specific frequency laser on an object in the real world; a process of calculating an incident light value, which is the value of the white board data at each of a plurality of wavelengths that includes the wavelength of the specific frequency laser, from white board data, which is the result of photographing a white board with a multispectral camera; a measurement light calculation unit that calculates a measurement value, which is the value of the multispectral camera data at each of the plurality of wavelengths that includes the wavelength of the specific frequency laser, from multispectral camera data, which is the result of photographing the object with the multispectral camera; and a process of calculating an albedo, which is the reflectance unique to the object, from the specific frequency laser reflectance, the incident light value, and the measurement value.

Images