Image processing device and image processing program

Abstract

An image processing device according to one aspect of the present invention comprises an input unit, a dictionary creation unit, and a densification unit. The input unit receives a specific frequency laser reflection intensity which is a measurement result of an object in a real world by a specific frequency laser sensor using a specific frequency laser, and multispectral camera data which is an imaging result of the object by a multispectral camera and includes a wavelength band of the specific frequency laser. The dictionary creation unit creates, from the inputted specific frequency laser reflection intensity and multispectral camera data, a database indicating a correspondence relationship between the specific frequency laser reflection intensity and the multispectral camera data. The densification unit performs densification calculation of the specific frequency laser reflection intensity on the basis of the created database to generate a densified specific frequency laser reflection intensity.

Description

Image processing device and image processing program

[0001] One aspect of the present invention relates to an image processing device and an image processing program.

[0002] Specific frequency laser sensors such as LiDAR (light detection and ranging) are widely used to obtain accurate distance information in autonomous driving, topographical surveys, urban measurement, and other applications. Recently, the usefulness of the reflected laser light intensity (LiDAR reflection intensity), obtained as a by-product of distance measurement using LiDAR sensors, has begun to be recognized. However, LiDAR reflection intensity is sparser (lower density) than general camera images, limiting its application.

[0003] There has been much research focused on improving the sparsity of LiDAR, addressing this issue through tasks such as depth completion and superresolution. However, most depth completion models are based on supervised learning, which requires the preparation of a large amount of sufficiently dense ground truth data, resulting in significant costs. Therefore, unsupervised learning and self-supervised learning models have been attracting attention in recent years.

[0004] Non-Patent Document 1 discloses an unsupervised LiDAR reflection intensity densification method that densifies LiDAR reflection intensities using an RGB image. Specifically, Non-Patent Document 1 discloses an unsupervised deep learning model that receives as input a LiDAR reflection intensity L to be compensated for, a LiDAR reflection intensity mask M, noise Z, and a corresponding RGB image I, and outputs compensated LiDAR reflection intensities. In Non-Patent Document 1, a loss function

[0005]

[0006] Furthermore, by using RGB images,

[0007]

[0008] By calculating this, we can expect further improvement in accuracy.

[0009] Sato Shogo, et al., "Unsupervised Intrinsic Image Decomposition with LiDAR Intensity", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023.

[0010] However, the densification technique disclosed in Non-Patent Document 1 may assign an incorrect LiDAR reflection intensity value to colorful objects whose reflection characteristics vary greatly depending on the wavelength.

[0011] This invention has been made in consideration of the above circumstances, and aims to provide a densification technology that makes it possible to precisely densify the reflection intensity of a specific frequency laser even for colorful objects whose reflection characteristics vary greatly depending on the wavelength.

[0012] In order to solve the above problems, an image processing device according to one embodiment of the present invention includes an input unit, a dictionary creation unit, and a densification unit. The input unit accepts specific-frequency laser reflection intensities, which are measurements of real-world objects using a specific-frequency laser sensor that uses a specific-frequency laser, and multispectral camera data, which are images of the object captured by a multispectral camera and contain the wavelength bands of the specific-frequency laser. The dictionary creation unit creates a database representing the correspondence between the specific-frequency laser reflection intensities and the multispectral camera data from the input specific-frequency laser reflection intensities and multispectral camera data. The densification unit performs densification calculations on the specific-frequency laser reflection intensities based on the created database to generate densified specific-frequency laser reflection intensities.

[0013] According to one aspect of the present invention, by utilizing data from a multispectral camera, it is possible to provide a densification technology that enables precise densification of specific frequency laser reflection intensity even for colorful objects whose reflection characteristics vary greatly depending on the wavelength.

[0014] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the software configuration of the image processing device. FIG. 3 is a diagram showing an example of hyperspectral camera data and LiDAR reflection intensity received by an input information acquisition unit in FIG. 2. FIG. 4 is a diagram showing an example of the contents of a spectral information database created by a dictionary creation unit in FIG. 2. FIG. 5 is a flowchart showing an example of image processing executed by a control unit of the image processing device. FIG. 6 is a schematic diagram showing the procedure of a densification calculation process. FIG. 7 is a diagram showing an example of densification LiDAR reflection intensity.

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] 1 and 2 are block diagrams showing an example of the hardware and software configurations of an image processing device 1 according to an embodiment of the present invention. The image processing device 1 may be a server computer connected to a network NW such as the Internet or a LAN (Local Area Network), or may be a user computer such as a personal computer (PC), smartphone, or tablet terminal that can be connected to the network NW.

[0017] The image processing device 1 includes a control unit 11, to which a storage unit having a program storage unit 12 and a data storage unit 13, a communication interface unit 14, and an input / output interface unit 15 are connected via a bus 16. In Fig. 1, "interface" is abbreviated as "IF."

[0018] The control unit 11 is a hardware processor such as a CPU (Central Processing Unit). For example, the CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded CPU. The control unit 11 may include multiple hardware processors.

[0019] Under the control of the control unit 11, the communication interface unit 14 transmits and receives information to and from other devices using a communication protocol defined by the network NW.

[0020] An input device 171 and an output device 172 are connected to the input / output interface unit 15. Note that the image processing device 1 does not necessarily have to be equipped with all of these devices, and some of the devices may be connected as external devices.

[0021] The input device 171 is used by a user of the image processing device 1 to input instructions and information necessary for the operation of the image processing device 1. If the image processing device 1 is a personal computer, the input device 171 may include, for example, a keyboard, a pointing device such as a mouse or a touchpad, etc. If the image processing device 1 is a smartphone or a tablet terminal, the input device 171 may include, for example, a touch panel or operation buttons arranged on the display screen of the output device 172. The input device 171 may also include a reader device for reading information from a recording medium storing various information such as programs and data. The input device 171 may also include a microphone for capturing audio, or a camera for capturing video, such as a webcam or a general video camera. The camera may also include a multispectral camera such as a hyperspectral camera. The input device 171 may also include a specific frequency laser sensor such as a LiDAR sensor.

[0022] The output device 172 includes a display, such as a liquid crystal monitor, an organic electroluminescence (EL) monitor, a projector screen, or a head-mounted display, that displays various information generated by the image processing device 1. The output device 172 may also include a speaker that transmits various information to the user by sound. Furthermore, the output device 172 may include a writer device that writes the information generated by the image processing device 1 to a recording medium.

[0023] The program storage unit 12 is configured by combining, for example, a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or an EEPROM (registered trademark) (electrically erasable programmable read-only memory) that can be written to and read from at any time, with a nonvolatile memory such as a read-only memory (ROM). The program storage unit 12 stores middleware such as an operating system (OS), as well as an image processing program, which is an application program required for operation as an image processing device according to an embodiment of the present invention. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0024] The data storage unit 13 is, for example, a combination of a nonvolatile memory that can be written to and read from as needed, such as an HDD, SSD, EEPROM, or memory card, and a volatile memory, such as a RAM (Random Access Memory), as storage media. The data storage unit 13 includes, in its storage area, the following storage units necessary for implementing one embodiment: a hyperspectral camera data storage unit 131, a LiDAR reflection intensity storage unit 132, a spectral information database 133, and a high-density LiDAR reflection intensity storage unit 134.

[0025] The hyperspectral camera data storage unit 131 stores hyperspectral camera data H, which is an example of multispectral camera data captured by a hyperspectral camera, which is an example of a multispectral camera. In the following description, the multispectral camera and the multispectral camera data are described as the hyperspectral camera and the hyperspectral camera data H.

[0026] The LiDAR reflection intensity storage unit 132 stores LiDAR reflection intensity L, which is an example of a specific frequency laser reflection intensity, as a measurement result of a LiDAR sensor, which is an example of a specific frequency laser sensor. In the following description, the specific frequency laser sensor and the specific frequency laser reflection intensity are described as the LiDAR sensor and the LiDAR reflection intensity L. Note that the hyperspectral camera data H contains the wavelength band of the laser frequency used by this LiDAR as the measurement wavelength region.

[0027] The spectral information database 133 is a database that indicates the correspondence between the LiDAR reflection intensity and the hyperspectral camera data. The spectral information database 133 will be described in detail later.

[0028] The high-density LiDAR reflection intensity memory unit 134 stores a high-density LiDAR reflection intensity L′, which is an example of a high-density specific frequency laser reflection intensity that has been densified by the control unit 11.

[0029] The control unit 11 includes, as processing function units according to an embodiment of the present invention, an input information acquisition unit 111, a dictionary creation unit 112, a densification calculation unit 113, and a densified LiDAR reflection intensity output unit 114. These processing function units 111 to 114 are all realized by causing a hardware processor of the control unit 11 to execute an image processing program according to an embodiment of the present invention, which is stored in the program storage unit 12.

[0030] The image processing program may be stored in advance in the program storage unit 12, or may be read out from a storage medium storing the image processing program when necessary and stored in the program storage unit 12. Alternatively, the image processing program may be downloaded from a program server (not shown) or the like and stored in the program storage unit 12. At least a part of the processing functions of at least one of the processing function units 111 to 114 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), instead of being realized by the image processing program and a hardware processor of the control unit 11.

[0031] The input information acquisition unit 111 accepts input information sent from an external device via the network NW received by the communication interface unit 14 or input information input from the input device 171 via the input / output interface unit 15, and stores the input information in the data storage unit 13. Specifically, the input information acquisition unit 111 accepts hyperspectral camera data, which is image data obtained by photographing a real-world object with a hyperspectral camera, and stores the hyperspectral camera data in the hyperspectral camera data storage unit 131. The input information acquisition unit 111 also accepts LiDAR reflection intensity, which is a measurement result of the LiDAR sensor on the object photographed by the hyperspectral camera, and stores the LiDAR reflection intensity in the LiDAR reflection intensity storage unit 132. In this way, the input information acquisition unit 111 is an example of an input unit that accepts specific frequency laser reflection intensity, which is a measurement result of a real-world object by a specific frequency laser sensor using a specific frequency laser, and multispectral camera data, which is a result of photographing the object with a multispectral camera and includes the wavelength band of the specific frequency laser.

[0032] 3 is a diagram showing an example of hyperspectral camera data H and LiDAR reflection intensity L received by the input information acquisition unit 111. This is an example of photographing and measuring real-world objects: shrubbery in front of a building wall and a road sign. In reality, the hyperspectral camera data H is color image data. Furthermore, as can be seen in the enlarged diagram below, the LiDAR reflection intensity L is a set of reflection intensity values ​​at measurement points where reflections were detected.

[0033] The dictionary creation unit 112 creates a database representing the correspondence between the hyperspectral camera data H stored in the hyperspectral camera data storage unit 131 and the LiDAR reflection intensity L stored in the LiDAR reflection intensity storage unit 132, as the spectral information database 133. FIG. 4 is a diagram showing an example of the contents of the spectral information database 133 created by the dictionary creation unit 112. The spectral information database 133 is a database in which values ​​of LiDAR reflection intensity L and values ​​of hyperspectral camera data H are registered in association with pixel numbers. The pixel number is an identification number for a record in the database. The dictionary creation unit 112 obtains pixel values ​​of the hyperspectral camera data H corresponding to an area containing a value in the LiDAR reflection intensity L, and checks whether there is a record in the spectral information database 133 in which the two values ​​match. The value of each pixel of the hyperspectral camera data H is spectral information consisting of values ​​of multiple wavelengths. The multiple wavelengths include the LiDAR wavelength band. If there is no matching record, the dictionary creation unit 112 issues a new pixel number and adds a record having the acquired value of LiDAR reflection intensity L and the value of hyperspectral camera data H to the spectrum information database 133. In this way, the dictionary creation unit 112 is an example of a dictionary creation unit that creates a database indicating the correspondence between specific frequency laser reflection intensity and multispectral camera data from the input specific frequency laser reflection intensity and multispectral camera data.

[0034] The densification calculation unit 113 performs a densification calculation of the LiDAR reflection intensity L based on the spectral information database 133 to generate a densified LiDAR reflection intensity L′, and stores the generated densified LiDAR reflection intensity L′ in the densified LiDAR reflection intensity storage unit 134. Specifically, using the spectral information database 133, the densification calculation unit 113 sets the LiDAR reflection intensity of the pixel number most similar to the spectrum of the pixel of the hyperspectral camera data H corresponding to the complement target region for which a value is not entered in the LiDAR reflection intensity L stored in the LiDAR reflection intensity storage unit 132 as the value of the complement target region. A specific calculation method used by the densification calculation unit 113 will be described later. In this way, the densification calculation unit 113 is an example of a densification unit that performs a densification calculation of a specific frequency laser reflection intensity based on the created database to generate a densified specific frequency laser reflection intensity.

[0035] The densified LiDAR reflection intensity output unit 114 transmits the densified LiDAR reflection intensity L' stored in the densified LiDAR reflection intensity memory unit 134 to another device via the network NW using the communication interface unit 14, or outputs it to the output device 172 via the input / output interface unit 15. For example, if the output device 172 is a display, an image representing the densified LiDAR reflection intensity L' is displayed. Furthermore, if the output device 172 is a writer device, the densified LiDAR reflection intensity L' is written to a recording medium as a file.

[0036] (Operation Example) Next, an operation example of the image processing device 1 configured as described above will be described. Fig. 5 is a flowchart showing an example of the processing procedure and processing content of image processing executed by the control unit 11 in accordance with the image processing program stored in the program storage unit 12 of the image processing device 1. The image processing shown in this flowchart is executed in response to a processing start instruction sent via the network NW received by the communication interface unit 14 or input from the input device 171 via the input / output interface unit 15.

[0037] 5 , the control unit 11 of the image processing device 1 first operates as an input information acquisition unit 111 to acquire input information transmitted from another device of any source via the network NW or information input from the input device 171 via the input / output interface unit 15, and stores the information in the data storage unit 13 (step S11). Specifically, if the input information is hyperspectral camera data H, the control unit 11 stores it in the hyperspectral camera data storage unit 131. If the input information is LiDAR reflection intensity L, the control unit 11 stores it in the LiDAR reflection intensity storage unit 132.

[0038] Then, the control unit 11 operates as the dictionary creation unit 112 and creates a database serving as a dictionary from the hyperspectral camera data H and the LiDAR reflection intensity L as the spectrum information database 133 (step S12). Generally, the LiDAR reflection intensity L is sparse (low density) compared to the hyperspectral camera data H. Therefore, in this embodiment, first, when a set T of pixels having a LiDAR reflection intensity value and a set F of pixels having no value are defined,

[0039]

[0040] For pixels that satisfy t and the value H of the hyperspectral camera data H t By associating the above, a spectrum information database 133 is created in the data storage unit 13.

[0041] Thereafter, the control unit 11 operates as the densification calculation unit 113 to calculate the densified LiDAR reflection intensity L′ (step S13). That is, the control unit 11 calculates the densified LiDAR reflection intensity L′ by subtracting the pixel L from the pixel L that has no value in the LiDAR reflection intensity L stored in the LiDAR reflection intensity storage unit 132. f is the pixel to be complemented, and this pixel to be complemented L f Calculate the optimal value for

[0042] 6 is a schematic diagram showing the procedure of the densification calculation process for performing this calculation. frepresents the characteristics of the object surface. In the hyperspectral camera data H, areas with high similarity in spectral information have high similarity in the object surface characteristics, and as a result, the values ​​of the LiDAR reflection intensity L are likely to be close. Therefore, the control unit 11 calculates the interpolation target pixel L in the spectral information database 133. f Spectral information, which is the pixel value of the hyperspectral camera data H corresponding to (0, 10, 12, ..., 103), is read from the hyperspectral camera data storage unit 131. In the example of FIG. 6 , spectral information of (0, 10, 12, ..., 103) is read from the hyperspectral camera data storage unit 131.

[0043] Then, the control unit 11 refers to the spectrum information database 133 using this spectrum information, and calculates the pixel t' having the most similar spectrum information using the following formula.

[0044]

[0045] Note that D is a function that expresses distance or similarity, and cosine similarity, for example, is used. In the example of Fig. 6, the spectral information most similar to the spectral information (0, 10, 12, ..., 103) is the spectral information of pixel number 1. Therefore, pixel number 1 is acquired as pixel t' having the most similar spectral information.

[0046] Therefore, the control unit 11 determines whether the interpolation target pixel L f The LiDAR reflection intensity L of this pixel number L in the spectral information database 133 t In the example of FIG. 6, the value of the pixel to be interpolated L f The LiDAR reflection intensity L of pixel number 1 is "100".

[0047] The control unit 11 stores the LiDAR reflection intensity L read from the LiDAR reflection intensity storage unit 132 in the high-density LiDAR reflection intensity storage unit 134. Then, the value of a pixel that has no value in the stored LiDAR reflection intensity L is stored in the high-density LiDAR reflection intensity storage unit 134. t ' value.

[0048] The control unit 11 performs such densification calculation processing as follows:

[0049]

[0050] This is repeated for all t to complement all defects in the input LiDAR reflection intensity L. As a result, the high-density LiDAR reflection intensity storage unit 134 stores a high-density LiDAR reflection intensity L' in which all defects have been complemented. Figure 7 is a diagram showing an example of this high-density LiDAR reflection intensity L'.

[0051] In addition, when using hyperspectral camera data H, there may be cases where an imbalance in the shade or illumination occurs. In such cases, not only are the spectra similar, but there are also pixels L that do not have a value for the LiDAR reflection intensity L. f In some cases, the value closer to the target location is more reliable. In such cases, you can embed the location information as follows:

[0052]

[0053] Here, λ is a constant that represents the weight and is changed depending on the application.

[0054] Once the high-density LiDAR reflection intensity L′ has been calculated in this manner, the control unit 11 operates as the high-density LiDAR reflection intensity output unit 114 and outputs the high-density LiDAR reflection intensity L′ stored in the high-density LiDAR reflection intensity memory unit 134 (step S14). Then, the control unit 11 ends the processing shown in this flowchart.

[0055] (Actions and Effects) As described above, in the image processing device 1 according to one embodiment, the input information acquisition unit 111 receives the LiDAR reflection intensity L, which is the measurement result of a real-world object using a LiDAR sensor, and the hyperspectral camera data H, which is the result of capturing an image of the object using a hyperspectral camera and contains the LiDAR wavelength band. The dictionary creation unit 112 creates a spectral information database 133 representing a correspondence between the LiDAR reflection intensity L and the hyperspectral camera data H, based on the input LiDAR reflection intensity L and the hyperspectral camera data H. Then, in the image processing device 1 according to one embodiment, the densification calculation unit 113 performs a densification calculation of the LiDAR reflection intensity L based on the created spectral information database 133, thereby generating a densified specific frequency laser reflection intensity. According to the image processing device 1 according to this embodiment, by utilizing the hyperspectral camera data H, it is possible to accurately densify the LiDAR reflection intensity L even for colorful objects whose reflection characteristics vary greatly depending on the wavelength.

[0056] In the image processing device 1 according to an embodiment, the dictionary creation unit 112 creates the spectral information database 133 using the areas containing the values ​​of the LiDAR reflection intensity L and the corresponding values ​​of the hyperspectral camera data H. Therefore, according to the image processing device 1 according to an embodiment, it is possible to create the spectral information database 133, which serves as a dictionary for acquiring the values ​​of the LiDAR reflection intensity L, using the values ​​of the hyperspectral camera data H as a key.

[0057] Furthermore, in the image processing device 1 according to one embodiment, the densification calculation unit 113 generates a densified LiDAR reflection intensity L′ by using the spectral similarity and the spectral information database 133 to calculate values ​​for areas where no value is entered in the input LiDAR reflection intensity L. Therefore, according to the image processing device 1 according to one embodiment, it is possible to calculate values ​​for areas where no value is entered in the LiDAR reflection intensity L based on the similarity between the values ​​of the hyperspectral camera data H corresponding to areas where no value is entered in the input LiDAR reflection intensity L and the values ​​of the hyperspectral camera data H stored in the spectral information database 133.

[0058] [Other Embodiments] The present invention is not limited to the above-described embodiment.

[0059] For example, the image processing device 1 may accept as input information multispectral camera data other than hyperspectral camera data H obtained using a multispectral camera other than a hyperspectral camera, and specific frequency laser reflection intensity other than LiDAR reflection intensity L obtained using a specific frequency laser sensor other than a LiDAR sensor, and output high-density specific frequency laser reflection intensity other than high-density LiDAR reflection intensity L'.

[0060] Furthermore, the image processing program stored in the program storage unit 12 may be transferred in a state where it is stored in the computer constituting the image processing device 1, or may be transferred in a state where it is not stored in the computer. In the latter case, the image processing program may be transferred via the network NW, or may be transferred in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium capable of storing a program and is readable by a computer, such as a CD-ROM or a memory card.

[0061] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0062] In short, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0063] DESCRIPTION OF SYMBOLS 1...Image processing device 11...Control unit 12...Program storage unit 13...Data storage unit 14...Communication interface unit 15...Input / output interface unit 16...Bus 111...Input information acquisition unit 112...Dictionary creation unit 113...Densification calculation unit 114...Densified LiDAR reflection intensity output unit 131...Hyperspectral camera data storage unit 132...LiDAR reflection intensity storage unit 133...Spectral information database 134...Densified LiDAR reflection intensity storage unit NW...Network

Claims

1. An image processing device comprising: an input unit that accepts specific frequency laser reflection intensities, which are the measurement results of a real-world object by a specific frequency laser sensor that uses a specific frequency laser, and multispectral camera data, which is the result of photographing the object with a multispectral camera and contains the wavelength band of the specific frequency laser; a dictionary creation unit that creates a database representing the correspondence between the specific frequency laser reflection intensities and the multispectral camera data from the input specific frequency laser reflection intensities and the multispectral camera data; and a densification unit that performs densification calculations on the specific frequency laser reflection intensities based on the created database to generate densified specific frequency laser reflection intensities.

2. The image processing device according to claim 1, wherein the dictionary creation unit creates the database using areas containing the values ​​of the specific frequency laser reflection intensity and the corresponding values ​​of the multispectral camera data.

3. An image processing device as described in claim 1 or 2, wherein the densification unit generates the densified specific frequency laser reflection intensity by using the spectral similarity and the database to calculate values ​​in areas where no value is entered in the input specific frequency laser reflection intensity.

4. An image processing program that causes a processor to execute the following processes: a process of accepting specific frequency laser reflection intensity, which is the measurement result of a real-world object by a specific frequency laser sensor using a specific frequency laser, and multispectral camera data, which is the result of photographing the object with a multispectral camera and contains the wavelength band of the specific frequency laser; a process of creating a database that indicates the correspondence between the specific frequency laser reflection intensity and the multispectral camera data from the input specific frequency laser reflection intensity and the multispectral camera data; and a process of performing a densification calculation of the specific frequency laser reflection intensity based on the created database to generate a densified specific frequency laser reflection intensity.

Images