Image processing device and image processing program

Abstract

This image processing device comprises: an acquiring unit that acquires a captured image obtained by imaging an object, and a reflection intensity distribution on the surface of the object; and an estimating unit that estimates at least one among a reflectance component that is not dependent on illumination light on the captured image and a shadow component that is dependent on illumination light on the captured image, on the basis of the captured image and the reflection intensity distribution.

Description

[Technical Field]

[0001] The disclosed technology relates to an image processing device and an image processing program. [Background technology]

[0002] In recent years, various computer vision tasks, including image recognition and semantic segmentation, have been widely used. In computer vision tasks, a challenge is that lighting conditions, such as whether it's in sunlight or shade, or whether it's sunrise or sunset, affect performance in object detection and other tasks.

[0003] Therefore, eigenimage decomposition techniques that estimate a reflectance component (albedo) independent of the irradiated light and a shading component (shade) dependent on the irradiated light, based on an image (i.e., decomposing it into a reflectance component and a shading component), are attracting attention. For example, Non-Patent Document 1 discloses a method using a VAE (Variational Autoencoder)-based unsupervised learning model as an eigenimage decomposition technique. Also, for example, Non-Patent Document 2 discloses a method using a supervised learning model as an eigenimage decomposition technique. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Liu, Yunfei, Yu Li, Shaodi You, and Feng Lu. "Unsupervised learning for intrinsic image decomposition from a single image." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 3248-3257, 2020. [Non-Patent Document 2] Zhu, Yongjie, Jiajun Tang, Si Li, and Boxin Shi. "DeRenderNet: Intrinsic Image Decomposition of Urban Scenes with Shape-(In) dependent Shading Rendering." In 2021 IEEE International Conference on Computational Photography (ICCP), pp. 1-11, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0005] The technology described in Non-Patent Document 1 is unsupervised learning, and therefore cannot learn to distinguish between shadows cast by an object and patterns formed on the object's surface, resulting in insufficient accuracy in estimating the reflectance component. Furthermore, the technology described in Non-Patent Document 2 is difficult to prepare for training because it is difficult to actually observe the reflectance component of an object, and attempting to generate training data through simulation leads to increased costs and decreased accuracy.

[0006] The disclosed technology was developed in view of the above points, and aims to provide an image processing device and an image processing program that can accurately estimate at least one of the reflectance component and the shading component in an image. [Means for solving the problem]

[0007] A first aspect of the present disclosure is an image processing apparatus comprising: an acquisition unit that acquires an image obtained by photographing an object and a reflectance intensity distribution on the surface of the object; and an estimation unit that estimates at least one of an illumination-independent reflectance component in the image and an illumination-independent shading component in the image based on the image and the reflectance intensity distribution.

[0008] A second aspect of this disclosure is an image processing program, which causes a computer to function as the image processing device described in the first aspect. [Effects of the Invention]

[0009] According to the disclosed technology, at least one of the reflectance component and the shading component in an image can be estimated with high accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example of the hardware configuration of an image processing device. [Figure 2] This is a block diagram showing the configuration of the image processing apparatus according to the first embodiment. [Figure 3] This figure shows the input and output of the intrinsic image decomposition model of the first embodiment. [Figure 4] This figure shows an example of the configuration of the intrinsic image resolution model of the first embodiment. [Figure 5] This figure shows an example of the configuration of the intrinsic image resolution model of the first embodiment. [Figure 6] This flowchart shows the image processing flow of the first embodiment. [Figure 7] This figure shows the input and output of the intrinsic image decomposition model of the second embodiment. [Figure 8] This figure shows an example of the configuration of the intrinsic image resolution model of the second embodiment. [Figure 9] This flowchart shows the image processing flow of the second embodiment. [Figure 10] This is a block diagram showing the configuration of the image processing apparatus according to the third embodiment. [Figure 11] This figure shows the input and output of the intrinsic image decomposition model of the third embodiment. [Figure 12] The flowchart below shows the image processing flow of the third embodiment. [Figure 13] This is a diagram showing the configuration of the image processing apparatus according to the fourth embodiment. [Figure 14]This figure shows the input and output of the intrinsic image decomposition model of the fourth embodiment. [Figure 15] This figure shows an example of the configuration of the intrinsic image resolution model of the fourth embodiment. [Figure 16] This flowchart shows the image processing flow of the fourth embodiment. [Modes for carrying out the invention]

[0011] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0012] <Overview of Embodiments> Before describing the details of each embodiment, a general overview common to all embodiments will be provided. Each embodiment relates to an intrinsic image decomposition technique that estimates a reflectance component (albedo) independent of the irradiated light and a shading component (shade) dependent on the irradiated light from an image obtained by photographing an object (i.e., decomposing the image into a reflectance component and a shading component).

[0013] Conventionally, particularly regarding the reflectance component, the accuracy of estimation has been insufficient due to the difficulty in distinguishing between shadows cast by objects and patterns formed on the object's surface, as well as the difficulty in preparing training data. Therefore, in each embodiment, the reflectance component and shadow component in the captured image are accurately estimated by using the reflectance intensity measured by a sensor capable of measuring actual reflectance intensity, such as a LiDAR (Light Detection and Ranging) sensor.

[0014] As an example, a LiDAR sensor measures the distance to an object based on the time it takes to receive the reflected light after irradiating the object with laser light, or based on the phase change between the emitted light and the received light. In a LiDAR sensor, multiple laser emitters are arranged vertically, and each emitter performs horizontal scanning (rotation) to measure the three-dimensional shape (three-dimensional coordinates) of an object. In addition, a LiDAR sensor can also measure the reflectivity at various points on the object's surface using the ratio of the emitted light to the received light. Since the reflectivity measured by this LiDAR sensor is the ratio of the emitted light to the received light, it is not affected by ambient light.

[0015] In this specification, "reflection intensity" refers to the measured value of reflection intensity (for example, a value corresponding to the ratio of light intensity between emitted and received laser light) measured by a sensor capable of measuring reflection intensity, such as a LiDAR sensor. Furthermore, in this specification, "reflection intensity distribution" refers to the distribution of measured reflection intensity values ​​at various positions on the surface of an object, measured by a sensor capable of measuring reflection intensity, such as a LiDAR sensor. Details of each embodiment will be described below.

[0016] <First Embodiment> Figure 1 is a block diagram showing the hardware configuration of the image processing device 10 of this embodiment. As shown in Figure 1, the image processing device 10 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage 14, input unit 15, display unit 16, and communication I / F (Interface) 17. Each component is connected to the others via a bus 19 so as to be able to communicate with each other.

[0017] The CPU 11 is a central processing unit that executes various programs and controls various parts. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an image processing program for estimating reflectance components and shadow components from captured images. The image processing program may be a single program or a group of programs composed of multiple programs or modules.

[0018] ROM12 stores various programs and data. RAM13 temporarily stores programs or data as a working area. Storage14 consists of a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs, including the operating system, and various data.

[0019] The input unit 15 includes, for example, a pointing device such as a mouse and a keyboard, and is used for various types of input. The display unit 16 is, for example, a liquid crystal display and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0020] Communication I / F17 is an interface for communicating with other devices, such as cameras and LiDAR sensors. This communication may utilize wired communication standards such as Ethernet® or FDDI, or wireless communication standards such as 4G, 5G, or Wi-Fi®.

[0021] Next, the functional configuration of the image processing device 10 will be described. Figure 2 is a block diagram showing an example of the functional configuration of the image processing device 10. As shown in Figure 2, the image processing device 10 comprises an acquisition unit 20, an estimation unit 22, and a learning unit 24 as its functional configuration. Each functional configuration is realized by the CPU 11 reading an image processing program stored in the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0022] The acquisition unit 20 acquires a captured image I obtained by photographing an object, and a reflectance intensity distribution L on the object's surface. The captured image I is taken, for example, by a digital camera. The reflectance intensity distribution L is measured by a LiDAR sensor as described above. Note that the captured image I and the reflectance intensity distribution L are obtained by photographing and measuring the same object, respectively.

[0023] The estimation unit 22 estimates at least one of the reflectance component R, which is independent of illumination light, and the shading component S, which is dependent on illumination light, in the captured image I, based on the captured image I and the reflectance intensity distribution L acquired by the acquisition unit 20. That is, the estimation unit 22 estimates at least one of the reflectance component R and the shading component S for each pixel of the captured image I.

[0024] Specifically, the estimation unit 22 estimates at least one of the reflectance component R and the shadow component S using the unique image decomposition model 30. Figure 3 shows the input and output of the unique image decomposition model 30 according to this embodiment. As shown in Figure 3, the unique image decomposition model 30 is a pre-trained model that takes the captured image I and the reflectance intensity distribution L as input and outputs the reflectance component R and the shadow component S. The estimation unit 22 obtains the reflectance component R and the shadow component S of the captured image I by inputting the captured image I and the reflectance intensity distribution L acquired by the acquisition unit 20 into the unique image decomposition model 30.

[0025] Figure 4 shows a detailed configuration example of the intrinsic image resolution model 30 according to this embodiment. As shown in Figure 4, the intrinsic image resolution model 30 encodes the reflectance component R into a domain-independent latent space using an encoder E cR and an encoder E that encodes the reflectance component R into a domain-dependent latent space p R and a decoder D that decodes the two encoder outputs into the reflectance component R R . Furthermore, the intrinsic image decomposition model 30 includes an encoder E that encodes a combination of the captured image I and the reflection intensity distribution L into a domain-independent latent space c I and an encoder E that encodes a combination of the captured image I and the reflection intensity distribution L into a domain-dependent latent space p I and a decoder D that decodes the outputs of encoder E c I and encoder E p I into the captured image I and the reflection intensity distribution L I . Furthermore, the intrinsic image decomposition model 30 includes an encoder E that encodes the shading component S into a domain-independent latent space c S and an encoder E that encodes the shading component S into a domain-dependent latent space p S and a decoder D that decodes the two encoder outputs into the shading component S S .

[0026] The estimation unit 22 inputs a combination of the captured image I and the reflection intensity distribution L to encoder E c I and encoder E p I , and inputs the output of encoder E c I and the output of encoder E p R to decoder D R , thereby obtaining an estimation result R' of the reflectance component R of the captured image I. In this case, the output of encoder E p R is obtained by mapping the output of encoder E p I to the domain of the reflectance component R. Also, the estimation unit 22 is configured such that encoder E cI Output and encoder E p S The output and decoder D S By inputting this into the encoder E, the estimated result S' of the shadow component S of the captured image I is obtained. p S The output is encoder E p I This is obtained by mapping the output to the domain of the shading component S.

[0027] The intrinsic image decomposition model 30 is preferably trained by unsupervised learning using a training captured image, a training reflectance distribution, a training reflectance image consisting of reflectance components, and a training shaded image consisting of shade components. Here, the training captured image, the training reflectance image, and the training shaded image may be independent of each other. That is, the source images for the training reflectance image and the training shaded image do not necessarily have to be the same, nor do they have to be the same as the training captured image. The training captured image and the training reflectance distribution are obtained by photographing and measuring the same object, respectively.

[0028] Such a unique image decomposition model 30 can, for example, be a VAE-based unsupervised learning model as described in Non-Patent Document 1 above, but is not particularly limited. For example, the unique image decomposition model 30 may be DNN (Deep Neural Network) based. Alternatively, supervised learning may be performed using multiple training data sets, including combinations of training captured images, training reflectance distributions, training reflectance images, and training shaded images. In this case, the combination of training captured images, training reflectance distributions, training reflectance images, and training shaded images is constructed using the correct training reflectance images and training shaded images for pairs of training captured images and training reflectance distributions obtained by photographing and measuring the same object.

[0029] Furthermore, for example, in Figure 4 above, the intrinsic image decomposition model 30 was explained using the case where the captured image I and the reflectance distribution L are combined before encoding, but it is not limited to this. The captured image I and the reflectance distribution L may also be combined after encoding.

[0030] Figure 5 shows a detailed example of the configuration of the unique image resolution model 30 in this case. As shown in Figure 5, the unique image resolution model 30 is encoder E c R And, encoder E p R And, Decoder D R This includes the following. Furthermore, the unique image decomposition model 30 encodes the captured image I into a domain-independent latent space using an encoder E. c I And encoder E encodes the captured image I into a domain-dependent latent space. p I And an encoder E that encodes the reflection intensity distribution L into a domain-independent latent space. c L And an encoder E encodes the reflection intensity distribution L into a domain-dependent latent space. p L And, including. Furthermore, the unique image resolution model 30 is encoder E c I and encoder E c L The combined output of, and encoder E p I and encoder E p L Decoder D decodes the combined output into the captured image I and the reflectance distribution L. I It includes. Furthermore, the unique image resolution model 30 includes encoder E c S And, encoder E p S And, Decoder D S This includes.

[0031] The estimation unit 22 processes the captured image I into an encoder E c I and encoder E pI input the reflection intensity distribution L into encoder E c L and encoder E p L input into encoder E c I output and the output of encoder E c L output and the output of encoder E p R output into decoder D R to obtain an estimation result R' of the reflectance component R of the captured image I. In this case, encoder E p R output is obtained by mapping the output of encoder E p I to the domain of the reflectance component R. Furthermore, the estimating unit 22, the output of encoder E c I output and the output of encoder E c L output and the output of encoder E p S output into decoder D S to obtain an estimation result S' of the shading component S of the captured image I. In this case, encoder E p S output is obtained by mapping the output of encoder E p I to the domain of the shading component S.

[0032] The estimating unit 22 may perform control to cause the display unit 16 to display at least one of the reflectance component R and the shading component S in the captured image I obtained as described above as an image.

[0033] The learning unit 24 trains the intrinsic image decomposition model 30 using a loss function that includes at least one of: an error between a result obtained by masking an estimated reflectance component R according to the reflection intensity distribution L and a result obtained by masking the reflection intensity distribution L according to the reflection intensity distribution L; and an error between a result obtained by masking an estimated shading component S according to the reflection intensity distribution L and a result obtained by masking a ratio of the captured image I to the reflection intensity distribution L according to the reflection intensity distribution L.

[0034] For example, the learning unit 24 may add the following term Tr1 relating to the reflectance component R as a term in the loss function. Tr1=λr×g(M×h(R),M×(L×α+β))

[0035] Here, λr is the weight relating to the estimation error of the reflectance component R. g(a,b) is a function relating to the errors of a and b, such as the Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). h(a) is a function that converts a to grayscale. α and β are parameters for aligning the scale and bias of the reflectance intensity distribution L obtained from a LiDAR sensor, etc., with the reflectance component R, and may be updated during the learning process.

[0036] M is a mask for reflectivity, where M is 1 if a reflectivity corresponding to a pixel exists, and M is 0 if there is no reflectivity distribution L corresponding to a pixel. Generally, the resolution of a LiDAR sensor is smaller than the resolution of a camera, and it is not always possible to project the reflectivity corresponding to all pixels in the captured image I, so such a mask is provided. That is, M×h(R) is the result of masking the grayscale reflectivity component R according to the reflectivity distribution L. M×(L×α+β) is the result of masking the reflectivity distribution L corrected by parameters α and β according to the reflectivity distribution L.

[0037] For example, the learning unit 24 may also add the term Ts1 relating to the shadow component S shown below as a term in the loss function. Ts1=λs×g(M×S,M×Iv / (L×γ+δ))

[0038] Here, λs is the weight relating to the estimation error of the shading component S. Iv is the pixel value (luminance value) of the captured image I. γ and δ are parameters for aligning the scale and bias of the reflectance intensity distribution L obtained from a LiDAR sensor, etc., with the pixel value Iv of the captured image I, and may be updated during the learning process. M×S is the result of masking the shading component S according to the reflectance intensity distribution L. M×Iv / (L×γ+δ) is the result of masking the ratio of the pixel value Iv of the captured image I with the reflectance intensity distribution L corrected by parameters γ and δ, according to the reflectance intensity distribution L.

[0039] Furthermore, depending on the imaging device used to capture image I, various image processing such as gamma correction may be applied to the RAW image to make it closer to human vision before it is output as captured image I. A RAW image is image data output directly from the image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), of the imaging device. In this case, if the pixel values ​​of captured image I, which have undergone various image processing, are used directly in calculating the loss, the accuracy of the learning process may decrease.

[0040] Therefore, the learning unit 24 may add at least one of the terms Tr2 relating to the reflectance component R and Ts2 relating to the shading component S, which take into account various image processing in the imaging device, as a term in the loss function. Tr2=λr×g(M×h(f(R)),M×(L×α+β)) Ts2=λs×g(M×S,M×f(Iv / (L×γ+δ)))

[0041] Here, f(a) is a function that restores the various image processing applied in the imaging device, such as the inverse transform function of gamma correction. That is, M×h(f(R)) is the result of restoring the reflectance component R, converting it to grayscale, and masking it according to the reflectance intensity distribution L. M×(L×α+β) is the result of correcting the reflectance intensity distribution L with parameters α and β, and then masking it according to the reflectance intensity distribution L. M×S is the result of masking the shading component S according to the reflectance intensity distribution L. M×f(Iv / (L×γ+δ)) is the result of restoring the ratio between the pixel value Iv of the captured image I and the reflectance intensity distribution L corrected with parameters γ and δ, and then masking it according to the reflectance intensity distribution L.

[0042] The learning unit 24 may also perform self-supervised learning on the unique image decomposition model 30.

[0043] Next, the operation of the image processing device 10 in this embodiment will be described. Figure 6 is a flowchart showing the flow of image processing by the image processing device 10. The CPU 11 reads the image processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it, thereby performing the image processing shown in Figure 6.

[0044] In step S10, the CPU 11, acting as the acquisition unit 20, acquires the captured image obtained by photographing the object. In step S12, the CPU 11, acting as the acquisition unit 20, acquires the reflection intensity distribution on the surface of the object.

[0045] In step S14, the CPU 11, acting as an estimation unit 22, estimates at least one of the reflectance component and the shadow component in the captured image acquired in step S10, based on the captured image and the reflectance intensity distribution acquired in steps S10 and S12. Once step S14 is completed, this image processing is finished.

[0046] As described above, in this embodiment, the image processing apparatus has an acquisition unit 20 that acquires an image obtained by photographing an object and the reflection intensity distribution on the surface of the object. The estimation unit 22 estimates at least one of the reflectance component in the image that is independent of the illumination light and the shading component in the image that is dependent on the illumination light, based on the image and the reflection intensity distribution. According to this embodiment, by using reflection intensity, at least one of the reflectance component and the shading component in the image can be estimated with high accuracy.

[0047] <Second Embodiment> Next, a second embodiment will be described. Regarding the image processing apparatus 10 according to the second embodiment, functions and configurations similar to those of the image processing apparatus 10 according to the first embodiment are denoted by the same reference numerals, and detailed descriptions are omitted. Note that the hardware configuration (see Figure 1) and functional configuration (see Figure 2) of the image processing apparatus 10 according to the second embodiment are the same as those of the first embodiment.

[0048] The acquisition unit 20 acquires the captured image I obtained by photographing the object, and the reflectance intensity distribution L on the surface of the object. The acquisition unit 20 also acquires the depth distribution D on the surface of the object. The depth distribution D can be determined from the three-dimensional coordinates measured by the LiDAR sensor, as described above.

[0049] The estimation unit 22 estimates at least one of the reflectance component R and the shadow component S in the captured image I based on the captured image I, the reflectance intensity distribution L, and the depth distribution D acquired by the acquisition unit 20. Specifically, the estimation unit 22 estimates at least one of the reflectance component R and the shadow component S using the unique image decomposition model 30 shown in Figure 7. Figure 7 shows the input and output of the unique image decomposition model 30 according to this embodiment. As shown in Figure 7, the unique image decomposition model 30 according to this embodiment is a pre-trained model that takes the depth distribution D as input in addition to the captured image I and the reflectance intensity distribution L, and outputs the reflectance component R and the shadow component S. The estimation unit 22 obtains the reflectance component R and the shadow component S of the captured image I by inputting the captured image I, the reflectance intensity distribution L, and the depth distribution D acquired by the acquisition unit 20 into the unique image decomposition model 30.

[0050] Figure 8 shows a detailed configuration example of the intrinsic image resolution model 30 according to this embodiment. As shown in Figure 8, the intrinsic image resolution model 30 encodes the reflectance component R into a domain-independent latent space using an encoder E c R And an encoder E encodes the reflectance component R into a domain-dependent latent space. p R Then, a decoder D decodes the two encoder outputs into the reflectance component R. R This includes the following. Furthermore, the unique image resolution model 30 encodes the combined captured image I, the reflectance distribution L, and the depth distribution D into a domain-independent latent space using an encoder E. c I Encoder E combines the captured image I, the reflection intensity distribution L, and the depth distribution D into a domain-dependent latent space. p I And, encoder E c I and encoder E p I Decoder D decodes the output into captured image I, reflection intensity distribution L, and depth distribution D. I This includes the following. Furthermore, the unique image resolution model 30 encodes the shading component S into a domain-independent latent space using an encoder E. cS And encoder E encodes the shading component S into a domain-dependent latent space. p S Then, a decoder D decodes the two encoder outputs into a shading component S. S This includes.

[0051] The estimation unit 22 combines the captured image I, the reflection intensity distribution L, and the depth distribution D and processes them into the encoder E c I and encoder E p I Input into encoder E c I Output and encoder E p R The output and decoder D R By inputting this into the encoder E, the estimated result R' of the reflectance component R of the captured image I is obtained. p R The output is encoder E p I This is obtained by mapping the output of the encoder E to the domain of the reflectance component R. In addition, the estimation unit 22 uses encoder E c I Output and encoder E p S The output and decoder D S By inputting this into the encoder E, the estimated result S' of the shadow component S of the captured image I is obtained. p S The output is encoder E p I This is obtained by mapping the output to the domain of the shading component S.

[0052] Next, the operation of the image processing device 10 in this embodiment will be described. Figure 9 is a flowchart showing the flow of image processing by the image processing device 10. The CPU 11 reads the image processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it, thereby performing the image processing shown in Figure 9.

[0053] In step S20, the CPU 11, acting as the acquisition unit 20, acquires the captured image obtained by photographing the object. In step S22, the CPU 11, acting as the acquisition unit 20, acquires the reflection intensity distribution on the surface of the object. In step S24, the CPU 11, acting as the acquisition unit 20, acquires the depth distribution on the surface of the object.

[0054] In step S26, the CPU 11, acting as an estimation unit 22, estimates at least one of the reflectance component and the shadow component in the captured image acquired in step S20, based on the captured image, reflectance intensity distribution, and depth distribution acquired in steps S20 to S24. Once step S26 is completed, the image processing is finished.

[0055] As described above, in the image processing apparatus according to this embodiment, the acquisition unit 20 further acquires the depth distribution on the surface of an object. The estimation unit estimates at least one of the reflectance component and the shading component based on the captured image, the reflectance intensity distribution, and the depth distribution. According to the image processing apparatus according to this embodiment, by also considering the depth distribution, the shading component in the image can be estimated with high accuracy.

[0056] <Third Embodiment> Next, a third embodiment will be described. Regarding the image processing apparatus 10 according to the third embodiment, functions and configurations similar to those of the image processing apparatus 10 according to the first and second embodiments are denoted by the same reference numerals, and detailed descriptions are omitted. Note that the hardware configuration (see Figure 1) of the image processing apparatus 10 according to the third embodiment is the same as that of the first embodiment.

[0057] Incidentally, depending on the imaging device used to capture image I, various image processing such as gamma correction may be applied to the RAW image to make it closer to human vision before it is output as captured image I. A RAW image is image data output directly from the image sensor, such as a CCD or CMOS, of the imaging device. In this case, if captured image I, which has undergone various image processing on the RAW image, is used directly to estimate the reflectance component R and the shadow component S, the accuracy of the estimation may decrease.

[0058] Therefore, the image processing device 10 according to this embodiment restores the captured image I to a RAW image before various image processing such as gamma correction is applied, and improves the accuracy of the estimation by estimating at least one of the reflectance component R and the shading component S based on the restored RAW image.

[0059] Referring to Figure 10, the functional configuration of the image processing apparatus 10 of this embodiment will be described. Figure 10 is a block diagram showing an example of the functional configuration of the image processing apparatus 10. As shown in Figure 10, the image processing apparatus 10 of this embodiment comprises, as a functional configuration, an acquisition unit 20, an estimation unit 22, a learning unit 24, and a restoration unit 26. Each functional configuration is realized by the CPU 11 reading an image processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0060] The acquisition unit 20 acquires the captured image I obtained by photographing an object, and the reflection intensity distribution L on the surface of the object. The restoration unit 26 restores the captured image I acquired by the acquisition unit 20 to a RAW image Ir. Any known method can be appropriately applied as the method for restoring to a RAW image Ir. For example, the restoration unit 26 may restore the captured image I to a RAW image Ir using an inverse transformation function of various image processing (e.g., gamma correction) applied to the RAW image output from the image sensor in the imaging device.

[0061] The estimation unit 22 estimates at least one of the reflectance component R and the shadow component S based on the RAW image Ir and the reflectance intensity distribution L restored by the restoration unit 26, instead of the captured image I. Specifically, the estimation unit 22 estimates at least one of the reflectance component R and the shadow component S using the unique image decomposition model 30 shown in Figure 11. Figure 11 shows the input and output of the unique image decomposition model 30 according to this embodiment. As shown in Figure 11, the unique image decomposition model 30 according to this embodiment is a pre-trained model that takes the RAW image Ir and the reflectance intensity distribution L restored from the captured image I by the restoration unit 26 as input and outputs the reflectance component R and the shadow component S. The estimation unit 22 obtains the reflectance component R and the shadow component S of the RAW image Ir by inputting the RAW image Ir and the reflectance intensity distribution L restored by the restoration unit 26 into the unique image decomposition model 30.

[0062] Next, the operation of the image processing device 10 in this embodiment will be described. Figure 12 is a flowchart showing the flow of image processing by the image processing device 10. The CPU 11 reads the image processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it, thereby performing the image processing shown in Figure 12.

[0063] In step S30, the CPU 11, acting as the acquisition unit 20, acquires the captured image obtained by photographing the object. In step S32, the CPU 11, acting as the acquisition unit 20, acquires the reflection intensity distribution on the surface of the object. In step S34, the CPU 11, acting as the restoration unit 26, restores the captured image acquired in step S30 into a RAW image.

[0064] In step S36, the CPU 11, acting as the estimation unit 22, estimates at least one of the reflectance component and the shading component in the RAW image based on the reflectance intensity distribution acquired in step S32 and the RAW image restored in step S34. Once step S36 is completed, the image processing is finished.

[0065] As described above, in the image processing apparatus according to this embodiment, the restoration unit 26 restores the captured image to a RAW image. The estimation unit 22 estimates at least one of the reflectance component and the shading component based on the RAW image restored by the restoration unit 26 and the reflectance intensity distribution, instead of the captured image. According to the image processing apparatus according to this embodiment, by using a RAW image, at least one of the reflectance component and the shading component in the image can be estimated with greater accuracy.

[0066] <Fourth Embodiment> Next, a fourth embodiment will be described. Regarding the image processing apparatus 10 according to the fourth embodiment, functions and configurations similar to those of the image processing apparatus 10 according to the first to third embodiments are denoted by the same reference numerals, and detailed descriptions are omitted. Note that the hardware configuration (see Figure 1) of the image processing apparatus 10 according to the fourth embodiment is the same as that of the first embodiment.

[0067] Referring to Figure 13, the functional configuration of the image processing apparatus 10 of this embodiment will be described. Figure 13 is a block diagram showing an example of the functional configuration of the image processing apparatus 10. As shown in Figure 13, the image processing apparatus 10 of this embodiment comprises, as a functional configuration, an acquisition unit 20, an estimation unit 22, a learning unit 24, and a correction unit 28. Each functional configuration is realized by the CPU 11 reading an image processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0068] The acquisition unit 20 acquires the captured image I obtained by photographing the object, and the reflection intensity distribution L on the surface of the object.

[0069] The correction unit 28 corrects the captured image I based on the reflection intensity distribution L acquired by the acquisition unit 20 and generates a corrected image Ic masked according to the reflection intensity distribution L. Specifically, the correction unit 28 generates the corrected image Ic by replacing the pixel value (luminance value) Iv of each pixel in the captured image I with the following pixel value Icv. Icv = M × (L × ε + ζ) / Iv

[0070] Here, ε and ζ are parameters used to align the scale and bias of the reflection intensity distribution L obtained from a LiDAR sensor, etc., with the pixel value Iv of the captured image I. M is a mask for the reflection intensity; M is 1 when a reflection intensity corresponding to a pixel exists, and M is 0 when a reflection intensity distribution L corresponding to a pixel does not exist. Generally, the resolution of a LiDAR sensor is smaller than the resolution of a camera, and it is not always possible to project the reflection intensity corresponding to all pixels in the captured image I, so such a mask is provided. That is, M × (L × ε + ζ) / Iv is the result of masking the ratio of the reflection intensity distribution L corrected by parameters ε and ζ to the pixel value Iv of the captured image I according to the reflection intensity distribution L.

[0071] The estimation unit 22 estimates at least one of the reflectance component R and the shadow component S based on the captured image I and the corrected image Ic generated by the correction unit 28. Specifically, the estimation unit 22 estimates at least one of the reflectance component R and the shadow component S using the unique image decomposition model 30 shown in Figure 14. Figure 14 shows the input and output of the unique image decomposition model 30 according to this embodiment. As shown in Figure 14, the unique image decomposition model 30 according to this embodiment is a pre-trained model that takes the corrected image Ic generated by the correction unit 28 and the captured image I as inputs and outputs the reflectance component R and the shadow component S.

[0072] As described above, due to the difference in resolution between the captured image I and the reflectance distribution L, the corrected image Ic may become a sparse image. Therefore, it is preferable that the intrinsic image decomposition model 30 according to this embodiment accepts both the corrected image Ic and the captured image I as inputs. The estimation unit 22 obtains the reflectance component R and the shading component S of the captured image I by inputting the captured image I and the corrected image Ic into the intrinsic image decomposition model 30.

[0073] Figure 15 shows a detailed configuration example of the unique image resolution model 30 according to this embodiment. As shown in Figure 15, the unique image resolution model 30 is an encoder E c R And, encoder E pR And, Decoder D R This includes the following. Furthermore, the unique image decomposition model 30 encodes the captured image I into a domain-independent latent space using an encoder E. c I And encoder E encodes the captured image I into a domain-dependent latent space. p I Encoder E encodes the corrected image Ic into a domain-independent latent space. c L Encoder E encodes the corrected image Ic into a domain-dependent latent space. p L This includes the following. Furthermore, the unique image resolution model 30 includes encoder E c I and encoder E c L The combined output of, and encoder E p I and encoder E p L Decoder D decodes the combined output into captured image I and corrected image Ic. L It includes. Furthermore, the unique image resolution model 30 includes encoder E c S And, encoder E p S And, Decoder D S This includes.

[0074] The estimation unit 22 processes the captured image I into an encoder E c I and encoder E p I Input the corrected image Ic to the encoder E c L and encoder E p L Input into encoder E c I Output and encoder E c L Output and encoder E p R The output and decoder D R By inputting this into the encoder E, the estimated result R' of the reflectance component R of the captured image I is obtained.p R The output is encoder E p I This is obtained by mapping the output of the encoder E to the domain of the reflectance component R. In addition, the estimation unit 22 uses encoder E c I Output and encoder E c L Output and encoder E p S The output and decoder D S By inputting this into the encoder E, the estimated result S' of the shadow component S of the captured image I is obtained. p S The output is encoder E p I This is obtained by mapping the output to the domain of the shading component S.

[0075] The learning unit 24 trains the eigenimage decomposition model 30 using a loss function that includes at least one of the following: the error between the result of masking the estimated reflectance component R according to the reflectance intensity distribution L and the result of masking the reflectance intensity distribution L according to the reflectance intensity distribution L, and the error between the result of masking the estimated shading component S according to the reflectance intensity distribution L and the result of masking the ratio of the corrected image Ic to the reflectance intensity distribution L according to the reflectance intensity distribution L.

[0076] For example, the learning unit 24 may add at least one of the following terms to the loss function: Tr3, which relates to the reflectance component R, and Ts3, which relates to the shadow component S. Tr3=λr×g(M×R,M×(L×α+β)) Ts3=λs×g(M×S,M×Icv / (L×γ+δ))

[0077] Here, M×R is the result of masking the reflectance component R according to the reflectance intensity distribution L. M×(L×α+β) is the result of masking the reflectance intensity distribution L corrected by parameters α and β according to the reflectance intensity distribution L. M×S is the result of masking the shading component S according to the reflectance intensity distribution L. M×Icv / (L×γ+δ) is the result of masking the ratio of the pixel value Icv of the corrected image Ic to the reflectance intensity distribution L corrected by parameters γ and δ according to the reflectance intensity distribution L.

[0078] The learning unit 24 may also perform self-supervised learning on the unique image decomposition model 30.

[0079] Next, the operation of the image processing device 10 in this embodiment will be described. Figure 16 is a flowchart showing the flow of image processing by the image processing device 10. The CPU 11 reads the image processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it, thereby performing the image processing shown in Figure 16.

[0080] In step S40, the CPU 11, acting as the acquisition unit 20, acquires an image obtained by photographing an object. In step S42, the CPU 11, acting as the acquisition unit 20, acquires the reflection intensity distribution on the surface of the object. In step S44, the CPU 11, acting as the correction unit 28, corrects the image acquired in step S40 based on the reflection intensity distribution L acquired in step S42, and generates a corrected image.

[0081] In step S46, the CPU 11, acting as the estimation unit 22, estimates at least one of the reflectance component and the shadow component in the captured image based on the captured image acquired in step S40 and the corrected image generated in step S44. Once step S46 is completed, the image processing is finished.

[0082] As described above, in this embodiment, the image processing apparatus has a correction unit 28 that corrects the captured image based on the reflectance intensity distribution. The estimation unit 22 estimates at least one of the reflectance component and the shading component based on the captured image corrected by the correction unit 28. According to this embodiment, by using a corrected image that reflects the reflectance intensity distribution, at least one of the reflectance component and the shading component in the image can be estimated with greater accuracy.

[0083] Furthermore, this disclosure may also be in the form of a combination of the above embodiments. For example, the third embodiment and the fourth embodiment may be combined so that the RAW image restored by the restoration unit 26 is corrected by the correction unit 28.

[0084] Furthermore, the various processes that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processes, such as ASICs (Application Specific Integrated Circuits). In addition, the various processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0085] Furthermore, although the above embodiments describe an embodiment in which the image processing program is pre-stored (installed) in the storage 14, the invention is not limited thereto. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be provided in a form downloaded from an external device via a network.

[0086] The technology of this disclosure can also be appropriately combined from the embodiments described above. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above descriptions of the configuration, function, operation, and effect are examples of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, within the scope that does not depart from the spirit of the technology of this disclosure, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the descriptions and illustrations shown above.

[0087] The following additional information is disclosed regarding the embodiments described above.

[0088] (Additional note 1) An image processing device, Memory and At least one processor connected to the memory, Equipped with, The aforementioned processor, The captured image obtained by photographing an object and the reflection intensity distribution on the surface of the object are acquired. Based on the captured image and the reflection intensity distribution, at least one of the reflectance component in the captured image that is independent of the illumination light and the shading component in the captured image that is dependent on the illumination light is estimated. An image processing device configured as follows.

[0089] (Additional note 2) A non-temporary storage medium that stores a program executable by a computer to perform image processing, The aforementioned image processing is, The captured image obtained by photographing an object and the reflection intensity distribution on the surface of the object are acquired. Based on the captured image and the reflection intensity distribution, at least one of the reflectance component in the captured image that is independent of the illumination light and the shading component in the captured image that is dependent on the illumination light is estimated. Non-transitory storage medium. [Explanation of Symbols]

[0090] 10 Image Processing Device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input section 16 Display section 17 Communication I / F 19 bus 20 Acquisition Department 22 Estimation part 24 Learning Department 26 Restoration Section 28 Correction section 30 Intrinsic Image Resolving Models

Claims

1. An acquisition unit that acquires an image obtained by photographing an object and the reflection intensity distribution on the surface of the object, An estimation unit that estimates at least one of a reflectance component in the captured image that is independent of the illumination light and a shadow component in the captured image that is dependent on the illumination light, based on the captured image and the reflectance intensity distribution. Equipped with, The aforementioned reflection intensity distribution is the distribution of measured reflection intensity values ​​at each position on the surface of the object, measured by a sensor capable of measuring actual reflection intensity values. The estimation unit takes the captured image and the reflection intensity distribution as input and estimates at least one of the reflectance component and the shadow component using a pre-trained model that is trained to output at least one of the reflectance component and the shadow component. Image processing device.

2. The trained model is trained by unsupervised learning using training images, training reflectance distributions, training reflectance images consisting of the reflectance components, and training shadow images consisting of the shadow components, or by supervised learning using multiple training data including combinations of training images, training reflectance distributions, training reflectance images consisting of the reflectance components, and training shadow images consisting of the shadow components. The image processing apparatus according to claim 1.

3. The system further includes a learning unit that trains the trained model using a loss function that includes at least one of the following: the error between the result of masking the estimated reflectance component according to the reflectance intensity distribution and the result of masking the reflectance intensity distribution according to the reflectance intensity distribution; and the error between the result of masking the estimated shadow component according to the reflectance intensity distribution and the result of masking the ratio of the captured image to the reflectance intensity distribution according to the reflectance intensity distribution. The image processing apparatus according to claim 1.

4. The acquisition unit acquires the depth distribution on the surface of the object, The estimation unit estimates at least one of the reflectance component and the shadow component based on the captured image, the reflectance intensity distribution, and the depth distribution. The image processing apparatus according to claim 1.

5. The system further includes a restoration unit that restores the captured image to a RAW image. The estimation unit estimates at least one of the reflectance component and the shadow component based on the RAW image restored by the restoration unit and the reflectance intensity distribution, instead of the captured image. The image processing apparatus according to claim 1.

6. The system further includes a correction unit that corrects the captured image based on the reflection intensity distribution and generates a corrected image masked according to the reflection intensity distribution. The estimation unit estimates at least one of the reflectance component and the shadow component based on the captured image and the corrected image. The image processing apparatus according to claim 1.

7. An image processing program for causing a computer to function as an image processing device according to any one of claims 1 to 6.

Images