Learning device, learning method, and learning program

Abstract

An acquisition unit of this learning device acquires, from a plurality of learning images observed in respectively different wavelength regions, point information for each pixel, which constitutes a five-dimensional array obtained by combining three-dimensional coordinates of a point present on a gaze line and a vector representing the direction of the gaze line. A learning unit of the learning device uses the point information acquired by the acquisition unit from the plurality of learning images and the plurality of learning images to learn a density estimation multi-layer perceptron that estimates a density estimation value, which is an estimated value of the probability of the existence of a substance which generates a signal at a point.

Description

[Technical Field]

[0001] The disclosed technology relates to a learning device, a learning method, and a learning program.

[0002] Objects reflect, absorb, or transmit electromagnetic waves of different wavelengths depending on the type and state of the material they comprise. In other words, differences in the reflection, absorption, or transmission characteristics of electromagnetic waves can be used to estimate the type of material that composes an object and the distribution of the intensity of certain characteristics. For example, in a certain observation wavelength range (specifically, the near-infrared region), a region with low chlorophyll absorption can be estimated as a region with a relatively high concentration of plant leaves. The wavelength range suitable for observation varies depending on the characteristic to be estimated or the material to be identified. For example, when the light source is sunlight or lighting, observations in the wavelength range from 0 μm to approximately 6 μm are suitable for obtaining information on reflection or absorption depending on the composition and structure of the material. On the other hand, when the light source is blackbody radiation from an object at a typical temperature on Earth, observations in the wavelength range from approximately 7.5 μm to approximately 10.5 μm are suitable for obtaining temperature and emissivity information. The peak wavelength of blackbody radiation, based on Wien's displacement law, is 10.6 μm at 0°C and 7.7 μm at 100°C.

[0003] Multispectral images record electromagnetic waves in multiple wavelength ranges. Specifically, the existence or characteristics of objects can be inferred by recording the reflection or absorption of electromagnetic waves in multispectral images. Multispectral images can be acquired using an image sensor or the like (e.g., a multispectral sensor or a hyperspectral sensor) that has multiple observation wavelength ranges. For observations on Earth other than meteorological fields, wavelength ranges where the influence of the atmosphere is relatively small and where the light transmittance is relatively high (so-called atmospheric windows) are relatively often used as the observation wavelength range of the sensor. On the other hand, for observations other than Earth, wavelength ranges such as X-rays are often used. Hereinafter, the observation wavelength range of the sensor will also be referred to as the observation band. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", 3 Aug 2020, URL [https: / / arxiv.org / pdf / 2003.08934.pdf]. [Non-patent document 2] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey "BARF: Bundle-Adjusting Neural Radiance Fields", 19 Aug 2021, URL [https: / / arxiv.org / pdf / 2104.06405.pdf]. Summary of the Invention [Problem to be solved by the invention]

[0005] One issue is that it is not easy to obtain multispectral images captured in observation bands appropriate for each application. Specifically, there are two methods for obtaining multispectral images: using a multispectral sensor and combining data acquired by sensors with different observable wavelength ranges. When obtaining multispectral images using a multispectral sensor, there are an unlimited number of combinations of observation wavelength ranges depending on the application. Therefore, in order to obtain the optimal multispectral sensor, it must be individually manufactured. Furthermore, if observations are made only in the wavelength range optimal for the application, only data with low versatility can be obtained.

[0006] On the other hand, when acquiring multispectral images by combining data acquired by sensors with different observable wavelength ranges, it is necessary to align the data sets. Specifically, it is necessary to fix the sensor position using a dedicated jig and to estimate the positional relationship of the data sets by photographing a calibration pattern that can be observed at any wavelength each time. The above-mentioned work requires specialized knowledge and advanced skills. Furthermore, it is necessary to assume the alignment of the data sets from the time of observation, making it difficult to freely combine data from an archive of previously acquired data.

[0007] The disclosed technology has been developed in consideration of the above points, and aims to make it possible to generate a multispectral image by combining data acquired by sensors with different observable wavelength ranges, without performing work related to alignment during observation. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a learning device that includes an acquisition unit that acquires point information for each pixel from a plurality of training images observed in different wavelength ranges, the point information being a five-dimensional array that combines three-dimensional coordinates of points on the line of sight with a vector representing the direction of the line of sight, and a learning unit that learns a density estimation multi-layer perceptron that estimates a density estimate, which is an estimate of the probability that a substance that generates a signal is present at the point, using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images.

[0009] A second aspect of the present disclosure is a learning method in which an acquisition unit acquires point information for each pixel from a plurality of training images observed in different wavelength ranges, the point information being a five-dimensional array combining three-dimensional coordinates of points on the line of sight with a vector representing the direction of the line of sight, and the learning unit trains a density estimation multilayer perceptron that estimates a density estimate, which is an estimate of the probability that a substance that generates a signal is present at the point, using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images.

[0010] A third aspect of the present disclosure is a learning program that causes a computer to function as each of the components that make up the learning device. [Effects of the Invention]

[0011] According to the disclosed technology, it is possible to generate a multispectral image by combining data acquired by sensors with different observable wavelength ranges, without performing work related to alignment during observation. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a learning device. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a learning device. [Figure 3] FIG. 1 is a schematic diagram showing an example of the configuration of a DNN in BARF. [Figure 4] FIG. 10 is a diagram showing an example of a processing flow when generating an image. [Figure 5] FIG. 10 is a diagram showing an example of a processing flow when learning a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the wavelength range of visible light, and external parameters of a sensor capable of observing visible light. [Figure 6] FIG. 10 is a diagram showing an example of a processing flow when learning a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the infrared wavelength range, and external parameters of a sensor capable of observing infrared light. [Figure 7] 10 is a flowchart showing the flow of a learning process. [Figure 8] FIG. 10 is a diagram showing an example of a processing flow for generating an image when a density estimation multilayer perceptron and a color estimation multilayer perceptron have a common multilayer perceptron. [Figure 9]FIG. 10 is a diagram showing an example of a processing flow for learning the density estimation multilayer perceptron, the color estimation multilayer perceptron related to the wavelength range of visible light, the common multilayer perceptron, and the external parameters of a sensor capable of observing visible light, when the density estimation multilayer perceptron and the color estimation multilayer perceptron have a common multilayer perceptron. [Figure 10] FIG. 10 is a diagram showing an example of a processing flow for learning the density estimation multilayer perceptron, the color estimation multilayer perceptron related to the infrared wavelength range, the common multilayer perceptron, and the external parameters of a sensor capable of observing infrared light, when the density estimation multilayer perceptron and the color estimation multilayer perceptron have a common multilayer perceptron. [Figure 11] FIG. 10 is a diagram showing an example of a processing flow when an image is generated from images captured by a visible light sensor, an infrared sensor, and an ultraviolet sensor. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0014] Fig. 1 is a block diagram showing the hardware configuration of a learning device 10 according to this embodiment. As shown in Fig. 1, the learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0015] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a learning program for executing the learning processing described below.

[0016] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0017] The communication I / F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0018] Next, the functional configuration of the learning device 10 will be described. Fig. 2 is a block diagram showing an example of the functional configuration of the learning device 10. As shown in Fig. 2, the learning device 10 includes, as its functional configuration, an acquisition unit 100, an estimation unit 101, a generation unit 102, and a learning unit 103. Each functional configuration is realized by the CPU 11 reading out a learning program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The learning device 10 also includes a learning database 200, a viewpoint database 201, a point information database 202, a DNN (Deep Neural Network) parameter database 203, an estimated value database 204, and a generated image database 205.

[0019] The acquisition unit 100 acquires point information relating to points on the line of sight from a plurality of training images observed in different wavelength ranges. The point information is a five-dimensional array combining the three-dimensional coordinates of points on the line of sight of the sensor with a vector representing the line of sight direction, and is information recorded for each pixel. If point information is not recorded, the acquisition unit 100 can generate point information for each pixel based on the image coordinates of each pixel and the camera parameters.

[0020] In this embodiment, the sensor observes an object whose transmittance is not approximately 100% or whose emissivity is not approximately 0% in any observed wavelength range, and whose presence signal is generated (e.g., brightness different from the background, or an edge is visible, etc.). Therefore, for example, images observed in the ultraviolet to thermal infrared wavelength range (i.e., approximately 0.2 μm to approximately 14.0 μm), which is a range that easily captures signals of reflection and absorption by surrounding objects, are used as training images. Specifically, in this embodiment, the wavelength ranges observed in the training images are the visible light wavelength range (i.e., approximately 0.4 μm to approximately 0.78 μm) and the near-infrared to thermal infrared wavelength range (i.e., approximately 0.78 μm to approximately 14 μm). Hereinafter, the range from near-infrared to thermal infrared is referred to as infrared. However, this example is not limiting. For example, in addition to these wavelength ranges, the ultraviolet wavelength range may also be used as the wavelength range in which the training images are observed. Furthermore, the wavelength range in which the learning images are observed may be the ultraviolet wavelength range and the infrared wavelength range.

[0021] Specifically, the acquisition unit 100 reads out the internal parameters and external parameters of the sensor when the training image was captured from the training database 200. The internal parameters include the focal length and lens distortion coefficient of the sensor. The internal parameters can be obtained by acquiring them from the sensor's specification information or by acquiring them as data from the sensor. The external parameters are parameters that represent the position and orientation of the sensor. The external parameters are automatically estimated as a set of sensor positions and orientations for one training image. The initial values ​​of the external parameters are random numbers, etc. The acquisition unit 100 then calculates the line of sight direction of each pixel from the internal parameters and external parameters, and acquires point information related to points on the line of sight.

[0022] The resolution and image size of the training images are set arbitrarily. The intervals and ranges of points related to the point information acquired by the acquisition unit 100 are set to NeRf (Neural Radiance Fields), BARF (Bundle-Adjusting Neural Radiance Fields), or values ​​used in general volume rendering. NeRf and BARF will be described in detail later. The acquisition unit 100 stores, in a point information database 202, point information acquired for a plurality of training images observed in different wavelength ranges.

[0023] The acquisition unit 100 also acquires point information for each pixel from an image captured by a plurality of sensors (hereinafter simply referred to as "sensors") each having a different observable wavelength range. In this embodiment, the plurality of sensors are a visible light sensor having an observation band in the wavelength range from about 0.4 μm to about 0.78 μm and an infrared sensor having an observation band in the wavelength range from about 0.78 μm to about 14 μm. However, this is not limiting. For example, in addition to these sensors, an ultraviolet sensor having an observation band in the wavelength range from about 0.2 μm to about 0.4 μm may also be applied as the plurality of sensors. Alternatively, an ultraviolet sensor and an infrared sensor may also be applied as the plurality of sensors.

[0024] Specifically, the acquisition unit 100 reads out viewpoint information from a viewpoint database 201. The viewpoint information is the internal and external parameters of the sensor when the captured image was captured. Then, the acquisition unit 100 calculates the line of sight direction of each pixel from the viewpoint information, and acquires point information related to points existing on the line of sight. Note that the resolution and image size of the captured image are set arbitrarily. Furthermore, the point interval and range related to the point information acquired by the acquisition unit 100 are set to values ​​used in NeRf, BARF, or general volume rendering. Then, the acquisition unit 100 stores the point information acquired from the captured image in a point information database 202.

[0025] The estimation unit 101 estimates a density estimation value by inputting the point information acquired from the captured image by the acquisition unit 100 into a density estimation multilayer perceptron. Specifically, the estimation unit 101 reads the point information acquired from the captured image by the acquisition unit 100 from the point information database 202. Then, the estimation unit 101 estimates a density estimation value by inputting the point information acquired from the captured image by the acquisition unit 100 into the density estimation multilayer perceptron. The density estimation value is an estimate of the probability that a substance that generates a signal is present. A high density estimation value indicates a high probability that a substance that generates a signal is present. Note that if a common object appears in all the captured images, it is assumed that the locations with high density estimation values ​​are common. Furthermore, parameters of the density estimation multilayer perceptron are stored in a DNN parameter database 203.

[0026] The density estimation multilayer perceptron is a multilayer perceptron (MLP) that estimates a density estimate. The density estimation multilayer perceptron is trained by a training unit 103.

[0027] Furthermore, the estimation unit 101 inputs point information acquired from an image captured by a sensor capable of observing a specific wavelength range to a color estimation multilayer perceptron associated with the specific wavelength range, thereby estimating a color estimation value for the specific wavelength range. Specifically, the estimation unit 101 inputs point information acquired from an image captured by a sensor capable of observing a specific wavelength range, among the point information stored in the point information database 202, to the color estimation multilayer perceptron associated with the specific wavelength range, thereby estimating a color estimation value. Parameters of the color estimation multilayer perceptron are stored for each wavelength range in the DNN parameter database 203.

[0028] The color estimation multi-layer perceptron is a multi-layer perceptron that estimates color estimates. The color estimation multi-layer perceptron is independent of the density estimation multi-layer perceptron. The color estimation multi-layer perceptron is trained by the training unit 103.

[0029] For example, the estimation unit 101 inputs point information acquired from an image captured by a visible light sensor into a color estimation multilayer perceptron related to the wavelength range of visible light, thereby estimating the RGB values ​​of the point related to the point information. Also, the estimation unit 101 inputs point information acquired from an image captured by an infrared sensor into a color estimation multilayer perceptron related to the wavelength range of infrared light, thereby estimating the reflection luminance, etc. of the point related to the point information in that wavelength range.

[0030] The estimation unit 101 stores density estimates and color estimates for each wavelength range in the estimate database 204. For example, if the specific wavelength range is that of visible light, the estimation unit 101 stores RGB values ​​and the density estimate σ in the estimate database 204. If the specific wavelength range is that of infrared light, the estimation unit 101 stores the reflected luminance and the density estimate σ in that wavelength range in the estimate database 204.

[0031] The generation unit 102 generates an image observed in a specific wavelength range based on the density estimate and the color estimate in the specific wavelength range estimated by the estimation unit 101. Specifically, the generation unit 102 reads the density estimate and the color estimate in the specific wavelength range from the estimate database 204. The generation unit 102 also reads the external parameters of a sensor capable of observing the specific wavelength range from the viewpoint database 201.

[0032] The generation unit 102 then performs volume rendering using the density estimate and the color estimate in a specific wavelength range to generate an image observed in the specific wavelength range. For example, if the specific wavelength range is the wavelength range of visible light, the generation unit 102 combines the RGB values ​​with the density estimate σ to obtain the RGB values ​​for each pixel. In this embodiment, a method used in volume rendering, such as a ray tracing method, is used to combine these values. The generation unit 102 then stores the generated image in a generated image database 205.

[0033] The training unit 103 trains a density estimation multilayer perceptron using point information acquired from multiple training images by the acquisition unit 100 and the multiple training images. The training unit 103 also trains a color estimation multilayer perceptron for a specific wavelength range using point information acquired from training images observed in the specific wavelength range and the training images observed in the specific wavelength range. The training unit 103 then stores the parameters of the trained density estimation multilayer perceptron and color estimation multilayer perceptron in the DNN parameter database 203.

[0034] Furthermore, the learning unit 103 optimizes the external parameters of the sensor capable of observing a specific wavelength range simultaneously with the learning process using the point information acquired from the learning image observed in the specific wavelength range and the learning image observed in the specific wavelength range. The learning unit 103 then stores the learned external parameters of the sensor capable of observing the specific wavelength range in the learning database 200.

[0035] For example, if point information acquired from training images observed in the visible light wavelength range and point information acquired from training images observed in the infrared wavelength range are stored in the point information database 202, the training unit 103 inputs these point information to a density estimation multi-layer perceptron.The training unit 103 then estimates a density estimate.The training unit 103 also inputs the point information acquired from training images observed in the visible light wavelength range to a color estimation multi-layer perceptron related to the visible light wavelength range, thereby estimating a color estimate in the visible light wavelength range.

[0036] The learning unit 103 then generates an image observed in the visible light wavelength range based on the estimated density value and the color estimate in the visible light wavelength range. The learning unit 103 then calculates the error between the generated image and a training image observed in the visible light wavelength range. The learning unit 103 then uses the error as loss to train a density estimation multilayer perceptron, a color estimation multilayer perceptron for the visible light wavelength range, and external parameters of a sensor capable of observing visible light. In other words, the learning unit 103 does not train a color estimation multilayer perceptron for the infrared wavelength range based on the error between the generated image and a training image observed in the visible light wavelength range. The differentiation of the external parameters, the loss calculation formula, and the optimization algorithm are set according to well-known methods related to BARF technology.

[0037] The learning unit 103 also inputs point information acquired from a training image observed in the infrared wavelength range into a color estimation multilayer perceptron for the infrared wavelength range, thereby estimating a color estimate for the infrared wavelength range. The learning unit 103 then generates an image observed in the infrared wavelength range based on the estimated density estimate and the color estimate for the infrared wavelength range. The learning unit 103 then calculates the error between the generated image and the training image observed in the infrared wavelength range. The learning unit 103 then uses the error as loss to train the density estimation multilayer perceptron, the color estimation multilayer perceptron for the infrared wavelength range, and the external parameters of the sensor capable of observing infrared light. In other words, the learning unit 103 does not train the color estimation multilayer perceptron for the visible light wavelength range based on the error between the generated image and the training image observed in the infrared wavelength range. The differentiation of the external parameters, the loss calculation formula, and the optimization algorithm are set according to known methods related to BARF technology.

[0038] The training unit 103 trains the color estimation multilayer perceptrons for each of the multiple wavelength ranges in a changeable order. Therefore, the training unit 103 may alternately train the color estimation multilayer perceptron for the infrared wavelength range and the color estimation multilayer perceptron for the visible light wavelength range. The training unit 103 may also preferentially train the color estimation multilayer perceptron for the wavelength range associated with the image with higher resolution.

[0039] In this embodiment, the training unit 103 trains the density estimation multilayer perceptron and the color estimation multilayer perceptron using BARF, which does not require the input of external parameters. However, this is not limiting. Among methods for generating an image in which a three-dimensional space is represented by density and color and observed from an arbitrary viewpoint, other methods that do not require the input of external parameters when training the DNN may be applied.

[0040] BARF is a method based on NeRf. In NeRf, volume rendering is performed on a vector field (radiance field) in which density and color are associated, thereby generating an image observed from an arbitrary viewpoint. Specifically, in NeRf, point information acquired from an image captured by a sensor is input to a DNN. The DNN then outputs the color and density of the points related to the input point information. NeRf then integrates the color and density output from the DNN to generate an image observed from an arbitrary viewpoint.

[0041] In NeRf, a DNN is trained using a set of images captured from multiple viewpoints, internal sensor parameters, and external sensor parameters as input. Specifically, point information acquired from the training images is input to the DNN during training. The DNN then outputs the color and density at the points related to the input point information. The DNN then trains by calculating the difference between the training image and an image generated by integrating the color and density output from the DNN.

[0042] On the other hand, unlike NeRf, BARF is a method that does not require input of external sensor parameters when training a DNN. This is because the external sensor parameters are automatically estimated during training. In other words, in BARF, a DNN is trained using a set of training images and internal sensor parameters as input.

[0043] Figure 3 shows the configuration of the DNN in BARF. As shown in Figure 3, point information acquired from an image captured by a visible light sensor is input to the multilayer perceptron included in the DNN. The multilayer perceptron then outputs color estimates and density estimates in the visible light wavelength range. An image observed in the visible light wavelength range is then generated from the color estimates and density estimates. Note that the numbers written on the multilayer perceptron in Figure 3 represent an example of the number of channels in each layer.

[0044] FIG. 4 shows a processing flow for generating an image in this embodiment. As shown in FIG. 4, the DNN according to this embodiment has a density estimation multilayer perceptron and a color estimation multilayer perceptron, which are separate components. The DNN according to this embodiment also has an independent color estimation multilayer perceptron for each wavelength range. Point information acquired from an image captured by a visible light sensor is input to a color estimation multilayer perceptron for the visible light wavelength range, thereby estimating a color estimate of the point associated with the point information in the visible light wavelength range. Point information acquired from an image captured by an infrared sensor is input to a color estimation multilayer perceptron for the infrared wavelength range, thereby estimating a color estimate of the point associated with the point information in the infrared wavelength range. Point information acquired from an image captured by a visible light sensor and point information acquired from an image captured by an infrared sensor are input to a density estimation multilayer perceptron, thereby estimating a density estimate. An image observed in the visible light wavelength range is then generated from the color estimate and density estimate in the visible light wavelength range. Additionally, an image observed in the infrared wavelength range is generated from the color and density estimates in the infrared wavelength range.

[0045] 5 shows a processing flow for training a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the wavelength range of visible light, and the extrinsic parameters of a sensor capable of observing visible light. As shown in Fig. 5, training unit 103 trains a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the wavelength range of visible light, and the extrinsic parameters of a sensor capable of observing visible light from a generated image observed in the wavelength range of visible light. In this case, training unit 103 does not train a color estimation multilayer perceptron related to the infrared wavelength range.

[0046] 6 shows a processing flow for training a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the infrared wavelength range, and external parameters of a sensor capable of observing infrared light. As shown in Fig. 6, training unit 103 trains a density estimation multilayer perceptron, a color estimation multilayer perceptron related to the infrared wavelength range, and external parameters of a sensor capable of observing infrared light from a generated image observed in the infrared wavelength range. In this case, training unit 103 does not train a color estimation multilayer perceptron related to the visible light wavelength range.

[0047] Next, we will explain the operation of the learning device 10. Figure 7 is a flowchart showing the flow of the learning process by the learning device 10. The learning process is performed by the CPU 11 reading out a learning program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0048] In step S11, the CPU 11 functions as the acquisition unit 100 to acquire point information relating to points present on the line of sight for each pixel from a plurality of learning images observed in different wavelength ranges.

[0049] Next, in step S12, the CPU 11, functioning as the learning unit 103, trains a density estimation multilayer perceptron using point information acquired from a plurality of training images and the plurality of training images.

[0050] Next, in step S13, the CPU 11, as the learning unit 103, trains a color estimation multilayer perceptron for a specific wavelength range using point information obtained from a training image observed in the specific wavelength range and the training image observed in the specific wavelength range.

[0051] Next, in step S14, the CPU 11, functioning as the learning unit 103, determines whether or not all color estimation multilayer perceptrons have been trained. If all color estimation multilayer perceptrons have been trained (step S14: YES), the CPU 11 proceeds to step S15. On the other hand, if all color estimation multilayer perceptrons have not been trained (step S14: NO), the CPU 11 returns to step S13.

[0052] Next, in step S15, the CPU 11 functions as the acquisition unit 100 to acquire point information for each pixel from the images captured by the multiple sensors.

[0053] Next, in step S16, the CPU 11, functioning as the estimation unit 101, inputs the point information acquired from the captured image in step S15 to a density estimation multilayer perceptron, thereby estimating a density estimate value.

[0054] Next, in step S17, the CPU 11, as the estimation unit 101, inputs point information obtained from an image captured by a sensor capable of observing a specific wavelength range into a color estimation multilayer perceptron related to the specific wavelength range, thereby estimating a color estimation value in the specific wavelength range.

[0055] Next, in step S18, the CPU 11 functions as the generating unit 102 to generate an image observed in a specific wavelength range based on the estimated density value and the estimated color value in the specific wavelength range.

[0056] Next, in step S19, the CPU 11, functioning as the generation unit 102, determines whether or not images in all wavelength ranges have been generated. If images in all wavelength ranges have been generated (step S19: YES), the CPU 11 ends this learning process. On the other hand, if images in all wavelength ranges have not been generated (step S19: NO), the CPU 11 returns to step S17.

[0057] One method for acquiring multispectral images is to use a hyperspectral sensor. Hyperspectral sensors have high wavelength resolution and can capture detailed spectral features. A single-wavelength sensor observes one band, a visible-light sensor observes three bands, a multispectral sensor observes approximately 20 bands, and a hyperspectral sensor observes approximately several hundred bands. However, high resolution reduces the amount of light received per pixel per image, resulting in a relative increase in noise. Furthermore, electromagnetic wave observations involve a trade-off between wavelength resolution, spatial resolution, and temporal resolution with respect to the signal-to-noise ratio (SNR). Because hyperspectral sensors have relatively high wavelength resolution, maintaining a minimum SNR requires lowering the spatial and temporal resolution. Reducing spatial resolution requires lowering the resolution. However, lowering the resolution requires capturing images at close range, otherwise fine spatial patterns will be lost. Reducing temporal resolution also requires increasing the exposure time. However, increasing the exposure time requires fixing the sensor and addressing the issue of moving objects being captured. Therefore, when a hyperspectral sensor is applied, there is a problem that the spatial resolution and the temporal resolution are reduced, or the noise is increased.

[0058] Furthermore, when combining data acquired by sensors with different observable wavelength ranges to obtain a multispectral image, it is necessary to align the data sets. This requires work to anticipate the alignment of the data sets from the time of observation, making it difficult to freely combine data from an archive of previously acquired data.

[0059] Therefore, the acquisition unit of the learning device according to this embodiment acquires point information for each pixel, which is a five-dimensional array combining the three-dimensional coordinates of points on the line of sight and a vector representing the line of sight direction, from multiple training images observed in different wavelength ranges. The learning unit of the learning device according to this embodiment then trains a density estimation multilayer perceptron, which estimates a density estimate, which is an estimate of the probability that a substance that generates a signal exists at a point, using the point information acquired from the multiple training images by the acquisition unit and the multiple training images. This makes it possible to generate a multispectral image by combining data acquired by sensors with different observable wavelength ranges, without performing any work related to alignment during observation.

[0060] <Modification> In the above embodiment, the density estimation multilayer perceptron and the color estimation multilayer perceptron do not share a common multilayer perceptron. However, this is not limiting. For example, as shown in FIG. 8, these multilayer perceptrons may share a common multilayer perceptron that shares a portion. That is, the common multilayer perceptron may be branched into a color estimation multilayer perceptron and a density estimation perceptron midway. In this case, it is possible to have a structure with eight layers of common multilayer perceptrons and one layer of color estimation perceptron for each sensor.

[0061] 9 shows a processing flow for training a density estimation multilayer perceptron, a color estimation multilayer perceptron for the wavelength range of visible light, a common multilayer perceptron, and external parameters of a sensor capable of observing visible light when these multilayer perceptrons have a common multilayer perceptron. As shown in Fig. 9, training unit 103 trains the density estimation multilayer perceptron, the color estimation multilayer perceptron for the wavelength range of visible light, the common multilayer perceptron, and external parameters of a sensor capable of observing visible light from a generated image observed in the wavelength range of visible light. In this case, training unit 103 does not train the color estimation multilayer perceptron for the wavelength range of infrared light.

[0062] 10 shows a processing flow for training the density estimation multilayer perceptron, the color estimation multilayer perceptron for the infrared wavelength range, the common multilayer perceptron, and the extrinsic parameters of the sensor capable of observing infrared light when the density estimation multilayer perceptron and the color estimation multilayer perceptron have a common multilayer perceptron. As shown in FIG. 10, the training unit 103 trains the density estimation multilayer perceptron, the color estimation multilayer perceptron for the infrared wavelength range, the common multilayer perceptron, and the extrinsic parameters of the sensor capable of observing infrared light from a generated image observed in the infrared wavelength range. In this case, the training unit 103 does not train the color estimation multilayer perceptron for the visible light wavelength range.

[0063] Furthermore, the learning device 10 may generate a new image from an image captured by a sensor of another wavelength range in addition to the visible light sensor and the infrared sensor. For example, a new image may be generated from an image captured by an ultraviolet sensor. FIG. 11 shows a processing flow for generating an image from an image captured by an ultraviolet sensor in addition to the visible light sensor and the infrared sensor. As shown in FIG. 11, point information acquired from the image captured by the ultraviolet sensor is input to a color estimation multilayer perceptron for the ultraviolet wavelength range, thereby estimating a color estimate of the point associated with the point information in the ultraviolet wavelength range. In addition, in a multilayer perceptron that has performed learning based on point information acquired from the image captured by the visible light sensor, point information acquired from the image captured by the infrared sensor, and point information acquired from the image captured by the ultraviolet sensor, point information generated for a desired angle of view is input to a density estimation multilayer perceptron to estimate a density estimate. Then, an image observed in the ultraviolet wavelength range is generated from the color estimate and density estimate in the ultraviolet wavelength range.

[0064] In the above embodiment, the learning device 10 includes the estimation unit 101 and the generation unit 102. However, this is not limiting. The learning device 10 does not necessarily have to include the estimation unit 101 and the generation unit 102.

[0065] Furthermore, the learning process executed by the CPU after reading the software (program) in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The learning process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0066] In the above embodiment, the learning program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0067] The following additional notes are provided regarding the above-described embodiments.

[0068] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: A five-dimensional array is obtained by combining three-dimensional coordinates of points on the line of sight and vectors representing the direction of the line of sight from a plurality of learning images observed in different wavelength ranges, and point information for each pixel is acquired; a density estimation multilayer perceptron that estimates a density estimation value, which is an estimate of the probability that a substance that generates a signal exists at the point, is trained using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images; A learning device configured as follows.

[0069] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to perform a learning process, The learning process includes: A five-dimensional array is obtained by combining three-dimensional coordinates of points on the line of sight and vectors representing the direction of the line of sight from a plurality of learning images observed in different wavelength ranges, and point information for each pixel is acquired; a density estimation multilayer perceptron that estimates a density estimation value, which is an estimate of the probability that a substance that generates a signal exists at the point, is trained using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images; Non-transitory recording media, including [Explanation of symbols]

[0070] 10 Learning Device 11 CPU 12 ROM 13 RAM 14. Storage 17 Communication I / F 19 Bus 100 Acquisition Department 101 Estimation part 102 Generation part 103 Learning Department 200 Learning Database 201 Viewpoint Database 202 point information database 203 DNN parameter database 204 Estimated Value Database 205 Generated Image Database

Claims

1. an acquisition unit that acquires point information for each pixel, the point information being a five-dimensional array that combines three-dimensional coordinates of points on the line of sight and vectors that represent the direction of the line of sight from a plurality of learning images observed in different wavelength ranges; a learning unit that learns a density estimation multilayer perceptron that estimates a density estimation value, which is an estimate of the probability that a substance that generates a signal exists at the point, using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images; A learning device comprising:

2. 2. The learning device of claim 1, wherein the learning unit estimates a color estimation value of the point in a specific wavelength range, and trains a color estimation multilayer perceptron independent of the density estimation multilayer perceptron using the point information acquired by the acquisition unit from the training image observed in the specific wavelength range and the training image observed in the specific wavelength range.

3. The learning device according to claim 2 , wherein the learning unit learns the color estimation multi-layer perceptron for each of a plurality of wavelength ranges in a changeable order.

4. The learning device according to claim 1 , wherein the plurality of training images are training images observed in a wavelength range of visible light and training images observed in a wavelength range from near-infrared to thermal infrared.

5. the acquisition unit acquires the point information for each pixel from images captured by a plurality of sensors each having a different observable wavelength range; an estimation unit that estimates the density estimation value by inputting the point information acquired from the captured image by the acquisition unit to the density estimation multilayer perceptron, and estimates the color estimation value in the specific wavelength range by inputting the point information acquired from the captured image captured by a sensor capable of observing the specific wavelength range to the color estimation multilayer perceptron related to the specific wavelength range; a generation unit that generates an image observed in the specific wavelength range based on the density estimation value and the color estimation value in the specific wavelength range estimated by the estimation unit; The learning device according to claim 2 , further comprising:

6. the acquisition unit acquires point information for each pixel, which is a five-dimensional array combining three-dimensional coordinates of points on the line of sight and vectors representing the direction of the line of sight, from a plurality of learning images observed in different wavelength ranges; a learning unit learning a density estimation multilayer perceptron that estimates a density estimation value, which is an estimate of the probability that a substance that generates a signal exists at the point, using the point information acquired from the plurality of training images by the acquisition unit and the plurality of training images; How to learn.

7. A learning program for causing a computer to function as each of the components constituting the learning device according to any one of claims 1 to 5.

Images