Image display device, method of generating a trained neural network model, and computer program

Abstract

An image display apparatus is provided that projects an image with high contrast using a phase modulation technique. The image display apparatus includes a trained neural network model that estimates a phase modulation distribution corresponding to an output target image; a phase modulation section that performs phase modulation on incident light based on the phase modulation distribution estimated by the trained neural network model; a luminance modulation section that performs luminance modulation on the phase-modulated light output from the phase modulation section; and a control section that outputs the phase-modulated and luminance-modulated incident light to a predetermined location.

Description

TECHNICAL FIELD

[0001] The technology disclosed herein (hereinafter referred to as "the present disclosure") relates to an image display device that projects an image with high contrast using a phase modulation technique, a method for generating a trained neural network model, and a computer program. BACKGROUND

[0002] Projection technology for projecting a video onto a screen has long been known, and has advantages such as allowing the same video to be projected to multiple people at the same time. Recently, improved content image quality, as exemplified by 4K or 8K resolution, has led to the emergence of projectors compatible with HDR (High Dynamic Range). For example, a projector is proposed that uses a spatial light modulator (SLM) to perform wavefront control on uniform light radiated from a light source to obtain a desired intensity distribution in which more light rays are concentrated in an area with high luminance, thereby achieving HDR (see PTL 1, for example). A free-form method is known as a method for obtaining a smooth phase distribution function suitable for driving a spatial light modulator.

[0003] [LIST OF CITATIONS]

[0004] [PATENT LITERATURE]

[0005] [PTL 1]

[0006] Japanese Patent Publication No. 2017-520022

[0007] [NON-PATENT LITERATURE]

[0008] [NPL 1]

[0009] R. T. Frankot and R. Chellapa, "A method for enforcing integrability in shape from shading algorithms", IEEE Trans. Pattern Anal. Mach. Intelligence, 10(4): 439-451, 1988.

[0010] [NPL 2]

[0011] A. Agrawal, R. Raskar, and R. Chellapa, "What is the range of surface reconstructions from a gradient field?", ECCV, 2006, pp. 578-591. SUMMARY

[0012] TECHNICAL PROBLEM

[0013] An object of the present disclosure is to provide an image display device that projects an image with high contrast using a phase modulation technique, a method for generating a trained neural network model, and a computer program.

[0014] Solution to the problem

[0015] A first aspect of the present disclosure is an image display device including: a trained neural network model that estimates a phase modulation distribution corresponding to an output target image; a phase modulation section that performs phase modulation on incident light with reference to the phase modulation distribution estimated by the trained neural network model; a luminance modulation section that performs luminance modulation on phase modulation light output from the phase modulation section; and a control section that outputs the incident light that has been subjected to phase modulation and luminance modulation to a predetermined position.

[0016] The trained neural network model is trained with reference to learning data including an intensity distribution input to the neural network model and a set of phase modulation distributions used as training data. Alternatively, the neural network model is trained in an unsupervised manner with reference to an error between an intensity distribution input to the neural network model and an intensity distribution calculated based on a ray optics model from the phase modulation distribution estimated by the neural network model.

[0017] Further, a second aspect of the present disclosure is a method for generating a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution, the method including: an input step of inputting an intensity distribution to the neural network model; an evaluation step of evaluating a phase modulation distribution estimated by the neural network model from the intensity distribution; and a learning step of training the neural network model with reference to a result of the evaluation.

[0018] Further, a third aspect of the present disclosure is a method for generating a trained neural network model that estimates a layout of rays corresponding to a target intensity distribution, the method including: an input step of inputting an intensity distribution to the neural network model; an evaluation step of evaluating a layout of rays estimated by the neural network model from the intensity distribution; and a learning step of training the neural network model with reference to a result of the evaluation.

[0019] Further, a fourth aspect of the present disclosure is a computer program described in a computer-readable format that executes, on a computer, a process for generating a trained neural network model that estimates a phase modulation distribution corresponding to a desired intensity distribution, the computer program causing the computer to function as: an input section that inputs an intensity distribution to the neural network model; an evaluation section that evaluates a result estimated by the neural network model from the intensity distribution; and a learning section that trains the neural network model with reference to a result of the evaluation.

[0020] Further, a fifth aspect of the present disclosure is a computer program described in a computer readable format, which executes a process on a computer for generating a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution, the computer program causing the computer to function as: an input section that inputs an intensity distribution to the neural network model; an evaluation section that evaluates a layout of light rays estimated by the neural network model from the intensity distribution; and a learning section that trains the neural network model with reference to a result of the evaluation.

[0021] The computer program according to each of the fourth and fifth aspects of the present disclosure is defined as a computer program described in a computer readable format to realize a predetermined process on a computer. In other words, by installing the computer program according to each of the fourth and fifth aspects of the present disclosure in a computer, a cooperative effect is exerted on the computer to allow an effect similar to that of the method for generating a trained neural network model according to each of the second and third aspects of the present disclosure to be produced.

[0022] Advantageous Effects of Invention

[0023] According to the present disclosure, it is possible to provide an image display device that uses a trained neural network model to estimate a phase modulation distribution corresponding to an output target image, to realize an increased contrast in real time, and to provide a method and a computer program for generating a trained neural network model.

[0024] It is to be noted that the effects described herein are merely illustrative, and the effects produced by the present disclosure are not limited to the effects described herein. Furthermore, the present disclosure can exert additional effects other than the above-described effects.

[0025] Other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description based on the embodiments described below and the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a diagram showing a configuration example of a projector 100.

[0027] Figure 2 is a diagram showing a configuration example of a projector 100 including a freeform estimation section using a trained neural network.

[0028] Figure 3A is a diagram depicting an example of a phase modulation distribution.

[0029] Figure 3B is a diagram showing a light ray grid having unequal grid intervals that is realized on an image plane as a result of a set of light rays (the set of light rays being incident with a uniform distribution parallel to the SLM) being bent by the phase modulation distribution shown in Figure 3A is a diagram showing a light ray grid having unequal grid intervals that is realized on an image plane as a result of a set of light rays (the set of light rays being incident with a uniform distribution parallel to the SLM) being bent by the phase modulation distribution shown in

[0030] Figure 4A To illustrate a plot of the sample values on the grid of light rays with unequal grid spacing shown in Figure 3B

[0031] Figure 4B To illustrate a plot of the sample values on the grid of light rays with equal grid spacing, the sample values obtained from the intensity distribution reproduced by the phase modulation distribution.

[0032] Figure 5 To depict a plot of the calculation of the intensity distribution from the phase modulation distribution and the calculation of the phase modulation distribution from the intensity distribution.

[0033] Figure 6 is a plot depicting a model of light propagation based on the free-form approach.

[0034] Figure 7 To depict an example of the correspondence between the grid on the SLM plane and the grid on the image plane.

[0035] Figure 8 is a plot depicting an example of the structure of the neural network 800 that estimates the free-form directly from the intensity distribution of the output target.

[0036] Figure 9 is a plot depicting a computational flow, where a neural network is used to estimate the grid layout of light rays from an intensity distribution, and where the free-form is then reconstructed by post-processing.

[0037] Figure 10 is a plot depicting an example of the structure of the neural network 901 that estimates the grid layout of light rays from the intensity distribution of the output target.

[0038] Figure 11 is a plot depicting the mechanism of supervised learning of the neural network 901 that estimates the grid layout of light rays from an intensity distribution.

[0039] Figure 12 is a plot showing a method for collecting learning data consisting of a set of intensity distributions and grid layouts.

[0040] Figure 13 is a plot depicting the mechanism of supervised learning of the neural network 800 that estimates the free-form from an intensity distribution.

[0041] Figure 14 is a plot depicting a method for collecting learning data comprising a set of intensity distributions and free-forms.

[0042] Figure 15 ​is a diagram depicting the mechanism of unsupervised learning of the neural network 901 that estimates the grid layout of light rays from the intensity distribution.

[0043] Figure 16 is a diagram depicting the flow in which the light ray optical model is used to calculate the reciprocal of the intensity distribution I(ux,uy) from the grid layout Δu(Δux,Δuy).

[0044] Figure 17 is a diagram depicting the flow in which the light ray optical model is used to calculate the reciprocal of the intensity distribution I(ux,uy) from the grid layout Δu(Δux,Δuy) and the intensity distribution I (incident) of the intensity distribution I(ux,uy).

[0045] Figure 18 is a diagram showing how the Reciprocal function is modified.

[0046] Figure 19 is a diagram depicting the mechanism of unsupervised learning of the neural network 901 that estimates the grid layout of light rays from the intensity distribution.

[0047] Figure 20 is a diagram depicting the flow in which the light ray optical model is used to calculate the reciprocal of the intensity distribution I(ux,uy) from the grid layout Δu(Δux,Δuy).

[0048] Figure 21 is a diagram depicting the flow in which the light ray optical model is used to calculate the reciprocal of the intensity distribution I(ux,uy) from the grid layout Δu(Δux,Δuy) and the intensity distribution I (incident) of the intensity distribution I(ux,uy).

[0049] Figure 22 is a diagram depicting the mechanism of unsupervised learning of the neural network 800 that estimates the freeform from the intensity distribution.

[0050] Figure 23 is a diagram depicting the mechanism of unsupervised learning of the neural network 800 that estimates the freeform from the intensity distribution.

[0051] Figure 24 is a diagram describing a configuration example of the illumination device 2400.

[0052] Figure 25 is a diagram describing a configuration example of the ToF sensor 2500. DETAILED DESCRIPTION

[0053] Referring to the drawings, embodiments of the present disclosure will be described in detail below in the following order.

[0054] A. Configuration of the projector

[0055] B. Phase modulation technique

[0056] C. Free-form method

[0057] D. Expression of intensity distribution in a ray optics model

[0058] E. Variation of free-form computation

[0059] F. Computational flow in the case of direct estimation of free-form from intensity distribution using a neural network

[0060] G. Computational flow in the case of estimation of a grid layout of rays from intensity distribution using a neural network, followed by reconstruction of free-form by post-processing

[0061] H. Learning method of neural network

[0062] H-1. Supervised learning

[0063] H-1-1. Supervised learning of neural network for estimation of grid layout of rays from intensity distribution

[0064] H-1-2. Supervised learning of neural network for estimation of free-form from intensity distribution

[0065] H-2. Unsupervised learning

[0066] H-2-1. Unsupervised learning of neural network for estimation of grid layout of rays from intensity distribution

[0067] H-2-2. Unsupervised learning of neural network for estimation of free-form from intensity distribution

[0068] I. Other application examples

[0069] A. Configuration of the projector

[0070] Figure 1 An example of the configuration of a projector 100 compatible with HDR and to which the present disclosure is applied is schematically described. The projector 100 includes a light source (not shown) that emits uniform light, a phase modulation panel 101, a luminance modulation panel 102, a magnifying optical system 103, and a control section 110 that controls the driving of the phase modulation panel 101 and the luminance modulation panel 102. Light having an intensity distribution as uniform as possible is incident on the phase modulation panel 101 from the light source. Uniform light refers to light having rays uniformly distributed. Light that has passed through the phase modulation panel 101 and the luminance modulation panel 102 is projected on a screen 104 via the magnifying optical system 103.

[0071] The projector incompatible with the HDR is configured so that the projector does not use the phase modulation panel 101 and uniform light from the light source is directly incident on the luminance modulation panel 102. The luminance modulation panel 102 includes a pattern based on an intensity distribution of a luminance component of a pixel in the target image, and light that has passed through the luminance modulation panel 102 is irradiated to the screen 104 to form a projection image of the target image. In this case, the control section 110 controls driving of the luminance modulation panel 102 (i.e., a luminance modulation distribution to be formed on the luminance modulation panel 102) with reference to the target image.

[0072] On the other hand, in the projector 100 compatible with the HDR, the phase modulation panel 101 applies phase modulation to the incident light to form a light ray distribution corresponding to the intensity distribution of the luminance modulation panel 102. Therefore, the light incident on the luminance modulation panel 102 has a light ray distribution in which more light rays are concentrated in a high luminance region, thereby realizing the HDR of the projection image. The projector 100 compatible with the HDR can project an image having high contrast with high energy efficiency.

[0073] In a case where the projector 100 is compatible with the HDR, the control section 110 controls driving of the phase modulation panel 101 and the luminance modulation panel 102 with reference to the target image. Specifically, the control section 110 controls formation of a phase modulation distribution in the phase modulation panel 101 and formation of a luminance modulation distribution in the luminance modulation panel 102 in accordance with the following procedure.

[0074] (Step 1) Setting of a target image

[0075] A target image to be finally displayed on the screen 104 is set.

[0076] (Step 2) Setting of a target intensity distribution in luminance modulation

[0077] A target intensity distribution to be formed on an exit surface 102a of the luminance modulation panel 102 by light exiting from the luminance modulation panel 102, which is light transmitted through the luminance modulation panel 102 in a case where the luminance modulation panel 102 is a transmissive type or light reflected from the luminance modulation panel 102 in a case where the luminance modulation panel 102 is a reflective type, is determined so that the target image is displayed on the screen 104. Figure 1

[0078] (Step 3) Setting of a target intensity distribution in phase modulation

[0079] In this processing step, a target intensity distribution of light exiting from the phase modulation panel 101 on a predetermined image surface is set. In a case where the phase modulation panel 101 is a transmissive type, the target intensity distribution is set so that the light exiting from the phase modulation panel 101 is light transmitted through the phase modulation panel 101. In a case where the phase modulation panel 101 is a reflective type, the target intensity distribution is set so that the light exiting from the phase modulation panel 101 is light reflected from the phase modulation panel 101. Figure 1 ​In the case of the HDR-compatible projector described in the above, the "predetermined image surface" corresponds to the incidence surface 102b of the luminance modulation panel 102. Therefore, in the processing step, in order to allow the above-described target intensity distribution to be formed on the exit surface 102a of the luminance modulation panel 102, the target intensity distribution to be formed on the incidence surface 102b of the luminance modulation panel 102 by the light exiting from the phase modulation panel 101 is determined.

[0080] (Step 4) Calculation of phase modulation distribution

[0081] A phase modulation distribution to be displayed on the phase modulation panel 101 is calculated, which causes the light exiting from the phase modulation panel 101 to form the target intensity distribution determined in the above-described step 3. In the present disclosure, a trained neural network model is used to calculate the phase modulation distribution corresponding to the target intensity distribution. The trained neural network model is mainly characterized by directly estimating a free-form phase modulation distribution that causes the ray density distribution of the light rays exiting from the phase modulation panel 101 to approximate the target intensity distribution on the incidence surface 102b of the luminance modulation panel 102. Details of this point are described later.

[0082] (Step 5) Calculation of luminance modulation distribution

[0083] With reference to the target image on the screen 104 and the phase modulation distribution calculated in the above-described step 4, a luminance modulation distribution to be displayed on the luminance modulation panel 102 is calculated. Specifically, when the phase modulation distribution calculated in the above-described step 4 is displayed on the phase modulation panel 101, the intensity distribution actually formed on the luminance modulation panel 102 by the light exiting from the phase modulation panel 101 is predicted, and with reference to the prediction result, a luminance modulation distribution is calculated, which causes the target image to approximate the image obtained by projecting the light exiting from the luminance modulation panel 102 on the screen 104.

[0084] B. Phase modulation technique

[0085] A phase modulation panel is an element that can modulate the phase of incident light for each pixel, and is also called a spatial light modulator (SLM). The phase modulation panel includes a liquid crystal phase modulation panel that modulates the phase by changing the refractive index of each pixel, a MEMS (Micro Electro Mechanical System) phase modulation panel that modulates the phase by displacing a micro-mirror in a direction perpendicular to the panel of each pixel, and the like.

[0086] A spatial light modulator can modulate the phase of a light wave to draw an image as non-uniformity of an intensity distribution generated on an image surface serving as a propagation destination. Figure 3A An example of a phase modulation distribution is depicted. Furthermore, Figure 3BIt is shown that a set of light rays incident with a uniform distribution parallel to the SLM pass through Figure 3A The phase modulation distribution shown is curved to achieve a light ray grid with unequal grid spacings on the image plane. In Figure 1 The projector 100 shown, as Figure 3A The phase modulation distribution shown can be formed by controlling the driving of the phase modulation panel 101. Figure 4A The sampling values on the light ray grid with unequal grid spacings depicted in Figure 3B are computed from the intensity distribution reproduced by the phase modulation distribution. Figure 4A The sampling values shown are computed as Figure 3B The density distribution of the light ray grid with unequal spacings shown. Furthermore, Figure 4B Sampling values on a grid with equal grid spacings are described, the sampling values being obtained from an intensity distribution reproduced by a phase modulation distribution. The sampling values are computed by resampling the intensity distribution depicted in Figure 4A with equal spacings on the grid. Note that the sampling values indicated in Figure 4B are computed from the phase modulation distribution shown in Figure 3A The reproduced image on the image plane is obtained as a reference light ray optics light ray density distribution from the intensity distribution shown in Figure 4A and Figure 4B The intensity distribution shown, and the actually reproduced image tends to be more blurred than the intensity distribution shown in Figure 4A and Figure 4B When a uniform light is incident on the phase modulation panel 101 with a non-uniform phase modulation distribution as shown in Figure 3A a light ray grid with unequal grid spacings as shown in Figure 3B is formed.

[0087] An intensity distribution reproduced from a certain phase modulation distribution can be computed using a light propagation model. For example, the reproduction of the intensity distribution depicted in Figure 3A and Figure 4A from the phase modulation distribution depicted in Figure 4B may be computed using a light propagation model. Note that an intensity distribution corresponding to an output target image (original image of the projected image on the screen 104) can be computed (however, the method for computing an intensity distribution corresponding to an output target image is not directly related to the object of the present disclosure, and is therefore not further described herein).

[0088] In the case where any image is to be reproduced, contrary to the above-described description, a phase modulation distribution achieving a certain intensity distribution needs to be computed. In other words, a phase modulation distribution for reproducing the intensity distribution described in Figure 4A and Figure 4B needs to be computed.

[0089] An exact solution of the phase modulation distribution from the intensity distribution is not possible, and therefore, an approximate phase modulation distribution is generally estimated. Methods for estimating the phase modulation distribution can include a computer-generated hologram (CGH) based on wave optics and a free-form method based on ray optics (see Figure 5 ). The CGH takes into account interference phenomena for phase estimation, and is therefore excellent in rendering when coherent light is used as an incident light source. However, the CGH requires discretization of the calculation region at a small sampling interval, requiring a large calculation time and high calculation cost. On the other hand, in the free-form method, the calculation is disadvantageously affected by interference of a coherent light source not taken into account, thereby preventing high-frequency components from being rendered in a fine manner. For example, Patent Literature 1 proposes an algorithm capable of calculating the phase modulation distribution at high speed using the free-form method. However, the algorithm requires repeated calculation and high calculation cost. Furthermore, the intensity distribution reproduced from the phase modulation distribution generated using the algorithm tends to have low contrast.

[0090] Therefore, in the present disclosure, a neural network is used to perform free-form estimation of the phase modulation distribution from the intensity distribution, thereby improving the contrast of the reproduced intensity distribution. In the case where the phase modulation technique is applied to the projector 100, it is necessary to perform phase estimation in real time. Furthermore, according to the present disclosure, the use of the neural network enables phase estimation to be performed at a higher speed, while also allowing the real-time characteristic to be satisfied.

[0091] C. Freeform method

[0092] The free-form method is a method for performing phase estimation using a ray optics model. Figure 6 A light propagation model based on the free-form method is schematically depicted. In Figure 6 , reference numeral 601 denotes an SLM surface of the phase modulation panel 101 (spatial light modulator), and reference numeral 602 denotes an image surface of the luminance modulation panel 102.

[0093] A group of light rays uniformly distributed in parallel to each other is incident on the SLM surface 601. In Figure 6 , the light rays are shown by arrows. The phase modulation panel 101 is of a liquid crystal type in which the refractive index varies with the pixel, and the exit side of the phase modulation panel 101 forms a smooth free surface corresponding to the phase modulation distribution 611. The wavefront of the light rays in the incident light ray group is continuously and smoothly distorted by the phase modulation distribution 611 on the SLM surface 601, and the direction of each light ray is bent in the normal direction of the wavefront. As a result, on the image surface 602, the light rays are distributed at unequal intervals, and this light ray density distribution forms the intensity distribution 612 of the light output from the luminance modulation panel 102.

[0094] The free-form method involves calculating the curved wavefront from the intensity distribution such that the light density distribution is as close as possible to the intensity distribution intended to be reproduced. However, the propagation of light is modeled as a refraction phenomenon that neglects diffraction. In the following description, "free-form" refers to the phase modulation distribution itself calculated based on such a light optical concept or the signal used as input to drive the phase modulation panel 101.

[0095] D. Expression of intensity distribution in ray optics model

[0096] Now, we will express the relationship between the phase modulation distribution P(x, y) on the SLM and the intensity distribution I on the image plane used as the propagation destination using ray optics. The incident light on the SLM is a plane wave, and we will now examine the equidistant grid points x = (x, y) perpendicularly incident on the SLM surface. T A set of rays. Using the phase modulation distribution P(x, y) on the SLM, the ray set penetrates the grid point u = (ux, uy) of the image plane at a distance f from the SLM. T It is represented by the following equation (1).

[0097] [Mathematical Expression 1]

[0098]

[0099] The displacement Δu between the equally spaced grid points on the SLM surface and the grid points on the image surface is expressed by equation (2).

[0100] [Mathematical Expression 2]

[0101]

[0102] That is, the displacement Δu between the phase modulation distribution P(x,y) and the grid points is a relationship between the scalar field and the gradient field.

[0103] refer to Figure 7 The square micro-region 701, surrounded by each grid point uniformly distributed on the SLM surface of the phase modulation panel 101 and the grid points adjacent to the aforementioned grid points, is examined. The micro-region 702 corresponding to the micro-region 701 on the SLM surface on the image surface of the brightness modulation panel 102 is a parallelogram. By controlling the wavefront of the SLM, the direction of each ray in the light group incident on the SLM is bent, forming an unequal spacing distribution on the image surface, so the grid points on the image surface are unequally spaced from each other. The area magnification m(ux, uy) of the micro-region 702 relative to the micro-region 701 is calculated using the calculation formula shown in equation (3) below.

[0104] [Mathematical Expression 3]

[0105]

[0106] The area magnification m(ux,uy) represented in the above equation (3) can be calculated from the grid point displacement (Δux, Δuy) as represented in the following equation (4) by using the above equation (2). T is represented.

[0107] [Equation 4]

[0108]

[0109] The electric field intensity I(ux,uy) at each grid point (ux,uy) on the image plane is calculated using the light density distribution 1 / m(ux,uy) as represented in the following equations (5) and (6).

[0110] [Equation 5]

[0111]

[0112] The above equation (5) corresponds to a ray optics model for calculating the intensity distribution I(ux,uy) from the phase modulation distribution P(x,y). Further, the above equation (6) corresponds to a ray optics model for calculating the intensity distribution I(ux,uy) from the grid layout Δu(Δux,Δuy) of the rays.

[0113] Here, I(ux,uy) represents the intensity value at the grid point (ux,uy) on the image plane corresponding to the grid point (x,y) on the SLM surface. Note that in numerical calculation, even in the case where the coordinates on the SLM surface are sampled at equi-spaced grid points (x,y), I(ux,uy) on the image plane corresponds to the sampled values obtained from the intensity distribution at grid points (ux,uy) which are not equi-spaced from each other. This is because the direction of each ray in the group of rays incident on the SLM is bent due to the wavefront control of the SLM, thereby forming a non-equally spaced distribution on the image plane.

[0114] In the case where the incident light having a non-uniform intensity distribution I (Incident) , the numerator 1 in the above equation (5) or (6) can be changed to the non-uniform weight I (Incident) as represented in the equations (7) or (8).

[0115] [Equation 6]

[0116]

[0117] As in the above equation (5), the above equation (7) corresponds to a ray optics model of calculating the intensity distribution I(ux,uy) from the phase modulation distribution P(x,y). Further, as in the above equation (6), the above equation (8) corresponds to a ray optics model of calculating the intensity distribution I(ux,uy) from the grid layout Au(Aux, Auy) of the rays.

[0118] Further, the grid point u on the image plane indicated in the above equation (1) can be deformed as in the following equation (9).

[0119] [Equation 7]

[0120]

[0121] In the above equation (9), "a" is a positive integer. The above equation (9) indicates that a certain phase modulation distribution P multiplied by "a" results in the same grid point layout of the rays on the surface at the projection distance multiplied by 1 / a as the original phase modulation distribution. Therefore, by assuming a certain projection distance to calculate the phase modulation distribution and then multiplying the phase modulation distribution by a constant, it is possible to change the projection distance of the reproduced image.

[0122] E. Variation of freeform calculation

[0123] In the present disclosure, a trained neural network is used to perform free-form estimation of a phase modulation distribution from an intensity distribution. The use of the neural network makes it possible to perform phase estimation at a higher speed while also allowing real-time characteristics to be satisfied. Further, by using the neural network to perform free-form estimation of a phase modulation distribution from an intensity distribution, the contrast of the reproduced intensity distribution increases.

[0124] Figure 2 A configuration example of a projector 100 configured to estimate a free form using a trained neural network is depicted. In addition to a phase modulation panel 101, a luminance modulation panel 102, a magnifying optical system 103, and a screen 104, Figure 2 The projector 100 shown in FIG. 1 includes a target intensity distribution calculation section 201, a free-form estimation section 202, and a luminance modulation distribution calculation section 203. The phase modulation panel 101, the luminance modulation panel 102, the magnifying optical system 103, and the screen 104 have been described, and detailed description thereof is omitted here.

[0125] The target intensity distribution calculation section 201 calculates a target intensity distribution in the phase modulation with reference to a target image to be finally displayed on the screen 104, that is, a target intensity distribution formed on the incident surface 102b of the luminance modulation panel 102 by light emitted from the phase modulation panel 101.

[0126] The free-form estimation section 202 estimates a phase modulation distribution, i.e., a free-form, to be displayed on the phase modulation panel 101 so that light exiting from the phase modulation panel 101 forms the target intensity distribution calculated by the target intensity distribution calculation section 201. In the present disclosure, the free-form estimation section 202 estimates the free-form using a trained neural network model. The trained neural network model is mainly characterized by directly estimating a free-form phase modulation distribution that brings the light ray density distribution exiting from the phase modulation panel 101 close to the target intensity distribution on the incident surface 102b of the brightness modulation panel 102, which will be described later in detail. The free-form estimation section 202 is, for example, incorporated in the control section 110 shown in FIG. 1. Figure 1

[0127] The brightness modulation distribution calculation section 203 calculates a brightness modulation distribution to be displayed on the brightness modulation panel 102 with reference to the target image on the screen 104 and the free-form estimated by the free-form estimation section 202. Specifically, when the free-form estimated by the free-form estimation section 202 using the trained neural network model is displayed on the phase modulation panel 101, the intensity distribution actually formed on the brightness modulation panel 102 by light exiting from the phase modulation panel 101 is predicted, and with reference to the prediction result, a brightness modulation distribution is calculated that brings the target image close to an image obtained by projecting the emitted light from the brightness modulation panel 102 on the screen 104. Here, the brightness modulation distribution calculation section 203 calculates the brightness modulation distribution using the free-form because it is necessary to take into account the simulation result for determining what kind of intensity distribution is actually formed on the incident surface 102b of the brightness modulation panel 102 when the free-form is displayed on the phase modulation panel 101. The brightness modulation distribution calculation section 203 is, for example, incorporated in the control section 110 shown in FIG. 1. Figure 1

[0128] Changes related to the calculation of the free-form will be described below, which are applied by the free-form estimation section 202.

[0129] (1) Changes related to the free-form calculation flow

[0130] (1-1) Direct estimation of the free-form from the intensity distribution of the output target using a neural network.

[0131] (1-2) Estimation of the grid layout of light rays from the intensity distribution of the output target using a neural network. The grid layout of light rays is data equivalent to the phase modulation distribution (the phase modulation distribution is a scalar field, while the grid layout is a gradient field corresponding to the scalar field). In post-processing, the phase modulation distribution is reconstructed from the estimated grid layout of light rays.

[0132] (2) Changes related to the learning method of the neural network​​

[0133] (2-1) Supervised learning

[0134] (2-1-1) Training a neural network that estimates a grid layout on an SLM surface in a supervised manner.

[0135] (2-1-2) Training a neural network that directly outputs a freeform in a supervised manner.

[0136] (2-2) Unsupervised learning

[0137] Training a neural network in an unsupervised manner by using a ray optics model as a guide.

[0138] (2-2-1) In the case of training a neural network that estimates a grid layout on an SLM surface in an unsupervised manner:

[0139] (2-2-1-1) The output of the ray optics model includes an intensity distribution.

[0140] (2-2-1-2) The output of the ray optics model is the inverse of an intensity distribution.

[0141] (2-2-2) In the case of training a neural network that directly outputs a freeform in an unsupervised manner:

[0142] (2-2-2-1) The output of the ray optics model includes an intensity distribution.

[0143] (2-2-2-2) The output of the ray optics model includes the inverse of an intensity distribution.

[0144] Methods for calculating a freeform in each variation will be described in turn below.

[0145] F. Calculation flow in case of direct estimation of freeform from intensity distribution using neural network

[0146] Figure 8 An example of the structure of a neural network 800 that directly estimates a freeform from an intensity distribution output by a target is depicted. The neural network 800 is designated to receive a normalized intensity distribution as input, such that the average intensity value is some constant (e.g., 1) and output a freeform (a signal input to the phase modulation panel 101 for driving). Note that in the following description, unless otherwise noted, the intensity distribution input to the neural network is intended to mean a normalized intensity distribution.

[0147] For example, in the case where the calculation resolution is M x N, the input to the neural network 800 can be an M x N single-channel array, and the output of the neural network 800 can be an M x N single-channel array, as Figure 8Alternatively, the input to the neural network 800 and the output from the neural network 800 can be one-dimensional, and the input can be an MN x 1 vector and the output is an MN x 1 vector. The network structure of the hidden layers of the neural network 800 is not particularly limited to any type (the network structure can be a convolution type or a fully connected type).

[0148] G. Calculation flow in case of estimation of grid layout of rays from intensity distribution using neural network, followed by reconstruction of freeform through post-processing Figure 9

[0149] Figure 10 A computational flow is depicted, in which the information related to the grid layout of the light is estimated from the intensity distribution of the output target using the neural network 901, and in which the phase modulation distribution is reconstructed by post-processing with reference to the estimated grid layout of the light.

[0150] The information related to the grid layout of the light output from the neural network 901 can be the displacement Au (Δux, Δuy) between the equidistant grid points (x, y) on the SLM surface and the corresponding grid points (ux, uy) on the image plane or the grid points (ux, uy) on the image plane corresponding to the grid points (x, y) on the SLM surface. The information related to the grid layout of the light is described as the displacement Au (Δux, Δuy) of the grid points (x, y) hereinafter, and the displacement Au (Δux, Δuy) is referred to as the grid layout hereinafter.

[0151] As described in the above item D, the free-form (or the phase modulation distribution P(x, y)) and the grid layout Au are related to the scalar field and the gradient field. As a result, the post-processing section 902 that calculates the free-form from the grid layout Au (Δux, Δuy) output from the neural network 901 can use an algorithm that reconstructs a surface from the gradient field (see NPL 1 and NPL 2, for example).

[0152] Figure 10 An example of the structure of the neural network 901 that estimates the grid layout of the light from the intensity distribution of the output target is depicted. The neural network 901 is specified to receive the intensity distribution as the input and output the grid layout of the light.

[0153] For example, as H. Learning method of neural networkAs shown, in a case where the calculation resolution is M x N, the input of the neural network 901 can be an M x N single-channel array, and the output of the neural network 901 can be an M x N double-channel array (each channel represents a grid point displacement (Δux, Δuy)). Alternatively, the input of the neural network 901 and the output of the neural network 901 can be made one-dimensional, and the input can be an MN x 1 vector, and the output can be a 2MN x 1 vector. The network structure of the hidden layer of the neural network 901 is not particularly limited to any type (the network structure can be a convolution type or a fully connected type).

[0154] H-1. Supervised learning

[0155] H-1-1. Supervised learning of neural network for estimation of grid layout of rays from intensity distribution

[0156] Figure 11

[0157] Figure 10 The mechanism of supervised learning of the neural network 901 (see Figure 12 ) that estimates a grid layout of a light ray from an intensity distribution is depicted. Learning data of a set composed of an intensity distribution (data input to the neural network 901) and a grid layout (training data) is collected in advance. When the intensity distribution of the input data corresponding to the learning data is input to the neural network 901, a predicted value Δu (predict) (Δux (predict) ,Δuy (predict) ) of the grid layout is output from the neural network 901. A loss function based on an error between the grid layout predicted by the neural network 901 and the grid layout of the training data is defined, and the neural network 901 is trained using backpropagation (error backpropagation) in a manner of minimizing the loss function.

[0158] The loss function used serves as an indicator of an error between the grid layout Δu (predict) (Δux (predict) ,Δuy (predict) ) predicted by the neural network 901 and the grid layout Δu(Δux, Δuy) of the training data. The loss function used can be, for example, a mean square error (MSE) L MSE indicated in the following equation (10), or a mean absolute error (MAE) L MSE indicated in the following equation (11).

[0159] [Math. 8]

[0160]

[0161] [Math. 9]

[0162]

[0163] In the above equations (10) and (11), n is an index representing each individual data sample in a mini-batch selected from the learning data at each learning step, and k is an index representing a sheet for array data or vector data. In addition, N is the size of the mini-batch.

[0164] The learning data including the set of intensity distribution and raster layout is preferably composed of a large amount of data. The learning data can be created using any freeform calculation algorithm or by first setting freeforms at random and collecting the set of raster layout and intensity distribution from freeform calculation. In either method, the above equations (6) or (8) related to the ray optics model can be used to calculate the intensity distribution from the raster layout. In addition, as explained in item D, the intensity distribution I(ux,uy) calculated according to the ray optics model has sampled values of the grid points (ux,uy) on the image plane with unequal intervals from each other, so it is necessary to re-sample the intensity distribution I(ux,uy) at the grid points on the image plane with equal intervals to obtain the input data.

[0165] H-1-2. Supervised learning of neural network for estimation of freeform from intensity distribution An example of a method for collecting learning data including the set of intensity distribution and raster layout is depicted.

[0166] An optional freeform calculation algorithm 1202 is used to calculate a freeform 1203 from a pre-collected output target intensity distribution 1201. Then, gradient field calculation is performed on the calculated freeform 1203 to calculate a raster layout 1204.

[0167] Depending on the freeform calculation algorithm 1202 used to calculate the freeform 1203, the correspondence in the set of output target intensity distribution 1201 and raster layout 1204 can deviate significantly from the physically correct correspondence. As a result, the set of output target intensity distribution 1201 and raster layout 1204 is not desirable as learning data.

[0168] As such, the intensity distribution 1206 is calculated from the raster layout 1204 using the ray optics model 1205 indicated in the above equations (6) or (8). In addition, the intensity distribution 1206 calculated by the ray optics model 1205 includes sampled values of the grid points (ux,uy) on the image plane with unequal intervals, so the intensity distribution 1207 is determined by re-sampling at the grid points (x,y) on the image plane with equal intervals. Then, the set of re-sampled intensity distribution 1207 (input data) and raster layout 1204 (training data) is determined as the learning data for the neural network 901.

[0169] Figure 13

[0170] Figure 8 The mechanism of supervised learning of the neural network 800 (see Figure 14 ) that estimates a freeform from an intensity distribution is depicted. Learning data including a set of intensity distributions (data input to the neural network 800) and grid layouts (training data) is collected in advance. When an intensity distribution of input data corresponding to the learning data is input to the neural network 800, a predicted value of a freeform is output from the neural network 800. A loss function based on an error between the freeform predicted by the neural network 800 and the freeform of the training data is defined, and the neural network 800 is trained using backpropagation in a manner that minimizes the loss function.

[0171] The loss function used serves as an indicator of an error between the freeform P (predict) predicted by the neural network 800 and the freeform P of the training data. The loss function used can be, for example, a mean square error (MSE) L MSE indicated in Equation (12) below, or a mean absolute error (MAE) L MSE indicated in Equation (13).

[0172] [Equation 10]

[0173]

[0174] [Equation 11]

[0175]

[0176] The learning data including a set of intensity distributions and grid layouts preferably includes a large amount of data. The learning data can be generated by using any freeform calculation algorithm or by first randomly setting a freeform and collecting a set of the freeform and an intensity distribution calculated from the freeform. In either method, the above-described Equation (5) or (7) related to a ray optics model can be used to calculate an intensity distribution from a freeform.

[0177] H-2. Unsupervised learning An example of a method for collecting learning data including a set of intensity distributions and freeforms is depicted.

[0178] A freeform 1403 is calculated from a pre-collected output target intensity distribution 1401 using an optional freeform calculation algorithm 1402.

[0179] Depending on the free-form computation algorithm 1402 used to compute the free form 1403, the correspondence in the set of output target intensity distribution 1401 and free form 1403 deviates significantly from the physically correct correspondence. As a result, the set of output target intensity distribution 1401 and free form 1403 is not expected to be used as learning data.

[0180] Therefore, the ray optical model 1404 shown in equation (5) or (7) above is used to calculate the intensity distribution 1405 from the free form 1403. Furthermore, the intensity distribution 1405 calculated by the ray optical model 1404 includes sampled values ​​of grid points (ux, uy) that are not equidistant from each other on the image plane. Therefore, the intensity distribution 1406 is determined by resampling the equally spaced grid points (x, y) on the image plane. Then, the set of the resampled intensity distribution 1406 (input data) and the free form 1403 (training data) is determined as the learning data for the neural network 800.

[0181] H-2-1. Unsupervised learning of neural network for estimation of grid layout of rays from intensity distribution

[0182] Using ray optics models as guidance allows for the training of neural networks in an unsupervised manner.

[0183] Figure 15

[0184] Possibly, there are methods where the output from the ray optics model used as a guide includes an intensity distribution, and methods where the output from the ray optics model used as a guide includes the inverse of the intensity distribution. First, a description of a method for training a neural network in an unsupervised manner will be given, where the output from the ray optics model used as a guide includes an intensity distribution.

[0185] Figure 10 The neural network 901 (see...) is described Figure 16 The unsupervised learning mechanism of the neural network 901 estimates the grid layout of the light rays based on the intensity distribution (the output from the ray optical model includes the intensity distribution).

[0186] When the intensity distribution I(x, y) corresponding to the input data of the learning data is input into the neural network 901, the predicted value Δu of the grid layout... (predict) (Δux (predict) ,Δuy (predict) The output is from neural network 901. Then, the ray optics model 1501 indicated in equation (6) or (8) above is used as the prediction value Δu from the grid layout. (predict) (Δux (predict) ,Δuy (predict) Reconstruction Intensity Distribution II reconstruct Guided by (ux,uy).

[0187] Intensity distribution I reconstructed from the ray optics model 1501 reconstruct (ux,uy) includes the sampled values for the unequally spaced grid points u (ux,uy) on the image plane. Therefore, the intensity distribution I(ux,uy) obtained by resampling the intensity distribution I(x,y) corresponding to the input data on the grid points u (ux,uy) estimated by the neural network 901 is determined. Then, a loss function of the error between the intensity distribution I(ux,uy) of the resampled input data and the predicted value Au (predict) (Δux (predict) ,Δuy (predict) ) of the grid layout output from the neural network 901 is calculated, and the neural network 901 is trained by backpropagation in a manner that minimizes the loss function. The loss function used can be used as an indicator of the error between the reconstructed intensity distribution I reconstruct (ux,uy) and the input intensity distribution I(ux,uy) obtained by resampling on the grid points u (ux,uy). reconstruct (ux,uy) and the input intensity distribution I(ux,uy) obtained by resampling on the grid points u (ux,uy).

[0188] Note that when the intensity distribution I reconstruct (ux,uy) reconstructed with the ray optics model 1501 is resampled at equally spaced grid points (x,y), error backpropagation is prohibited, and thus this method cannot be employed.

[0189] Figure 17 A flow of calculating the intensity distribution I(ux,uy) from the grid layout Au(Δux,Δuy) using the above equation (6) with respect to the ray optics model is shown. Furthermore, Figure 16 A flow of calculating the intensity distribution I(ux,uy) from the grid layout Au(Δux,Δuy) and the intensity distribution I (incident)) of the incident light on the phase modulation panel using the above equation (8) related to the ray optics model is described. When the predicted value Au (predict) (Δux (predict) ,Δuy (predict) ) of the grid layout is input to the calculation flow indicated in Figure 17 or Figure 16 , the reconstructed intensity distribution I reconstruct (ux,uy) is output.

[0190] Figure 17 and Figure 16 The difference operations and in Figure 17 and Figure 18g(·) in the above equation (13) represents an inverse function g(z) = 1 / z. The inverse is referred to as the reciprocal. When z = 0, the inverse function g(z) is discontinuous. Therefore, as depicted in Figure 15 the inverse function modified as shown in the following equation (14) is ideally used so that the difference coefficient is constant when z < ε (assuming ε is a normal number).

[0191] [Equation 12]

[0192]

[0193] In the above equation (14), it is desirable to set the normal number ε to a range from 0.01 to 0.1. Further, the ReLU function in the above equation (14) is defined by the following equation (15).

[0194] [Equation 13]

[0195]

[0196] In the learning framework as depicted in Figure 15 , the neural network 901 is trained to output a grid layout having a grid density distribution close to the density distribution in the input data. However, there is not necessarily a free form P(x, y) corresponding to the scalar field of the gradient field of the grid layout Δu (predict) (Δux (predict) ,Δuy (predict) ) output from the neural network 901. The reason is as follows: as a consistency in vector analysis, it is known that the rotation of the gradient of any scalar field is always zero (i.e., curl = 0), but Figure 9 the framework of learning as depicted in does not include a mechanism for establishing the consistency.

[0197] There is a condition that the free form P(x, y) corresponding to the scalar field is that the rotation field (rotation) of the grid layout Δu (predict) (Δux (predict) ,Δuy (predict) ) corresponding to the free form P(x, y) is zero everywhere. The rotation field of the grid layout Δu (predict) (Δux (predict) ,Δuy (predict) ) is shown in the following equation (16).

[0198] [Equation 14]

[0199]

[0200] Therefore, the neural network 901 needs to be trained to output a rotation of the gradient of the scalar field to be zero, i.e., a grid layout Δu (predict) (Δux (predict) ,Δuy (predict) ) in which a rotation field indicated by Δu reconstruct (ux,uy) in the above equation (16) is zero or close to zero

[0201] In addition to an indicator of an error between the reconstructed intensity distribution I (predict) (ux,uy) and an input intensity distribution I(ux,uy) obtained by resampling at the grid points u(ux,uy), a regularization term representing an average value of the magnitude of the component of the rotation field of the grid layout is introduced into the loss function. This makes it possible to reduce distortion of the reproduced image caused by post-reconstruction processing performed when reproducing a free form (see Figure 19 ). The regularization term related to the rotation field of the grid layout can be, for example, a mean square error defined by the following equation (17), a mean absolute error defined by the following equation (18), or the like.

[0202] [Equation 15]

[0203]

[0204] [Equation 16]

[0205]

[0206] Now, a description will be given of a method of training a neural network in an unsupervised manner, in which the output from a ray optics model used as a guide includes an inverse of an intensity distribution.

[0207] Figure 10 A mechanism of unsupervised learning of the neural network 901 (see Figure 15 ) that estimates a grid layout of rays from an intensity distribution (a case in which the output from a ray optics model includes an inverse of an intensity distribution) is depicted.

[0208] When an intensity distribution I(x, y) corresponding to input data of learning data is input to the neural network 901, a predicted value Δu (predict) (Δux (predict) ,Δuy (predict) ) of the grid layout is output from the neural network 901. Then, the ray optics model 1901 indicated in the above equation (6) or (8) is used as a guide to reconstruct an intensity distribution I (predict) (ux,uy) from the predicted value Δu (predict) (Δux (predict) ,Δuy reconstruct ) of the grid layout.

[0209] WithFigure 15 The learning scenario is similar, where the intensity distribution I(ux, uy) is determined by resampling the intensity distribution I(x, y) corresponding to the input data at grid point u(ux, uy). Then, the reciprocal of the intensity distribution I(ux, uy) of the resampled input data is calculated, and the predicted value Δu based on this reciprocal and the grid layout output from neural network 901 is calculated. (predict) (Δux (predict) ,Δuy (predict) Reconstructed intensity distribution I reconstruct The loss function is the error between the reciprocals of (ux, uy). The neural network 901 is trained via backpropagation in a manner that minimizes the loss function.

[0210] The loss function used can be used as the intensity distribution I of the reconstruction. reconstruct (ux,uy) is an indicator of the error between the input intensity distribution I(ux,uy) obtained by resampling at grid point u(ux,uy). Further, as in Figure 20 The learning scenario described above, in which regularization terms related to the rotation field of the grid layout (see equations (17) and (18) above) are expected to be added to the loss function.

[0211] Figure 21 The process of using equation (6) related to the ray optical model to calculate the reciprocal of the intensity distribution I(ux, uy) from the grid layout Δu(Δux, Δuy) is described. Furthermore, Figure 20 The above equation (8) is shown, which uses the grating layout Δu(Δux, Δuy) and the intensity distribution of the incident light on the phase modulation panel, which are related to the optical model of light. (incident)) The process for calculating the intensity distribution I(ux, uy). When the predicted value Δu of the grid layout... (predict) (Δux (predict) ,Δuy (predict) ) was input to Figure 21 or H-2-2. Unsupervised learning of neural network for estimation of freeform from intensity distribution When the calculation process is described in the figure, the reconstructed intensity distribution I reconstruct The reciprocal of (ux, uy) is output.

[0212] Figure 22

[0213] Possibly, there are methods where the output from the ray optics model used as a guide includes an intensity distribution, and methods where the output from the ray optics model used as a guide includes the inverse of the intensity distribution. First, a description of a method for training a neural network in an unsupervised manner will be given, where the output from the ray optics model used as a guide includes an intensity distribution.

[0214] Figure 8The neural network 800 (see...) is described Figure 15 The unsupervised learning mechanism of the neural network estimates the free form based on the intensity distribution (where the output from the ray optics model includes the intensity distribution).

[0215] When the intensity distribution I(x, y) corresponding to the input data of the learning data is input into the neural network 800, the neural network 800 outputs a free-form prediction value P. (predict) (x,y). Then, using the ray optics model 2301 indicated in equation (5) or (7) above as a guide to determine the free form P predicted by the neural network 800. (predict) (x,y) to reconstruct the intensity distribution I reconstruct (ux,uy).

[0216] Furthermore, the above equation (1) is used to obtain the free form P predicted by the neural network 800. (predict) (x,y) calculates the grid point u(ux,uy) formed by a set of light rays that are incident on the phase modulation panel 101 and then refracted by the free form and pass through the image surface of the brightness modulation panel 102.

[0217] like Figure 23 The learning scenario shown involves determining the intensity distribution I(ux, uy) obtained by resampling the intensity distribution I(x, y) corresponding to the input data at grid points u(ux, uy). Then, the intensity distribution I(ux, uy) based on the resampled input data is calculated along with the free-form predicted value P output from the neural network 800. (predict) Intensity distribution I reconstructed from (x,y) reconstruct The loss function is the error between (ux, uy). The neural network 800 is trained via backpropagation in a manner that minimizes the loss function. The loss function used can be used as the intensity distribution I of the reconstruction. reconstruct (ux,uy) is an indicator of the error between the input intensity distribution I(ux,uy) obtained by resampling at grid point u(ux,uy).

[0218] Note that the neural network 800 predicts the free form P(x, y) corresponding to the scalar field, thus preventing the problem of a lack of a scalar field that provides the grid layout as a gradient field. Therefore, the regularization term associated with the rotation field of the grid layout (see equations (17) and (18) above) does not need to be added to the loss function.

[0219] A description of a method for training a neural network in an unsupervised manner will now be given, wherein the output from the optical model of the ray used as a guide includes the inverse of the intensity distribution.

[0220] Figure 15A mechanism of unsupervised learning of the neural network 800 is described, which estimates the free-form from the intensity distribution (in the case where the output from the ray optics model includes the reciprocal of the intensity distribution).

[0221] When the intensity distribution I(x, y) of the input data corresponding to the learning data is input to the neural network 800, a predicted value P (predict) (x,y) of the free-form is output from the neural network 800. (predict) (x,y) predicted by the neural network 800. reconstruct (x,y) predicted by the neural network 800.

[0222] Further, the above equation (1) is used to calculate the grid point u(ux, uy) formed by a set of light rays incident on the phase modulation panel 101, then refracted by the free-form and passing through the image plane of the luminance modulation panel 102 from the free-form P (predict) (x,y) predicted by the neural network 800.

[0223] As I. Other application examples indicated in the case of learning, the intensity distribution I(ux, uy) obtained by resampling the intensity distribution I(x, y) corresponding to the input data on the grid point u(ux, uy) is determined. Then, the reciprocal of the resampled intensity distribution I(ux, uy) of the input data is calculated, and a loss function of the error between the reconstructed intensity distribution I (predict) (ux,uy) based on the reciprocal and the predicted value P reconstruct (x,y) output from the neural network 800 is calculated. The neural network 800 is trained by backpropagation in such a way that the loss function is minimized. The loss function used can be used as an indicator of the error between the reciprocal of the reconstructed intensity distribution I reconstruct (ux,uy) and the reciprocal of the input intensity distribution I(ux,uy) obtained by resampling on the grid point u(ux,uy).

[0224] Note that the neural network 800 predicts the free-form P(x, y) corresponding to a scalar field, and therefore the regularization term (see the above equations (17) and (18)) related to the rotational field of the grid layout does not need to be added to the loss function.

[0225] Figure 24

[0226] An example in which the free-form estimation technique using the trained neural network model according to the present disclosure is applied to an HDR-compatible projector is mainly described. The present disclosure is applicable to a lighting device that generally dynamically changes an intensity distribution.

[0227] Figure 1 An example of a layout in which the illumination device 2400 of the present disclosure is applied is schematically shown. The illumination device 2400 includes a light source (not shown) that radiates uniform light, a phase modulation panel 2401 that applies phase modulation to the uniform light from the light source to emit light onto an image plane 2403, and a control section 2402. The control section 2402 controls a phase modulation distribution to be displayed on the phase modulation panel 2401.

[0228] The image plane 2403 corresponds to Figure 24 The incident plane 102b of the brightness modulation panel 102 in the projector 100 shown. In Figure 24 In the case of the illumination device 2400 shown, the emitted light from the phase modulation panel 2401 is directly used as illumination light, and thus the image plane 2403 corresponds to the projection surface itself. Note that, in Figure 25 In the present embodiment, the illumination device 2400 is drawn in a simplified manner for ease of description, and before the phase modulation distribution is displayed on the phase modulation panel 2401, the phase modulation distribution is calculated and multiplied by a constant to allow the distance between the phase modulation panel 2401 and the image plane 2403 to be variable.

[0229] The control section 2402 controls the driving of the phase modulation panel 2401 with reference to a target image to be emitted by the illumination device 2400. Specifically, the control section 2402 controls the formation of the phase modulation distribution on the phase modulation panel 2401 in accordance with the following procedure.

[0230] (Step 1) Setting of a target image

[0231] A target image to be displayed on (in other words, to be illuminated) the image plane 2403 is set.

[0232] (Step 2) Setting of a target intensity distribution in phase modulation

[0233] A target intensity distribution of the emitted light from the phase modulation panel 2401 on the image plane 2403 is set.

[0234] (Step 3) Calculation of a phase modulation distribution

[0235] A phase modulation distribution to be displayed on the phase modulation panel 2401 is calculated, which causes the emitted light from the phase modulation panel 2401 to form the target intensity distribution determined in Step 2 described above. In the present disclosure, a trained neural network model is used to calculate the phase modulation distribution corresponding to the target intensity distribution. The trained neural network model directly estimates a free-form phase modulation distribution that causes the density distribution of the light rays emitted from the phase modulation panel 2401 to approximate the target intensity distribution. Details of the trained neural network model are as described above (see items C, D, and F to H described above).

[0236] ​ An example of a layout of a ToF (Time of Flight) sensor 2500 to which the illumination device according to the present disclosure is applied is described. The ToF sensor is a distance sensor based on distance measurement that measures a time taken for projected light to return to the original position after being reflected by an object, which is converted into a distance to the object. The depicted ToF sensor 2500 includes a light projection section 2510, a light reception section 2520, a signal processing section 2530, a target intensity distribution calculation section 2540, and a free form estimation section 2550.

[0237] The light projection section 2510, the light reception section 2520, and the signal processing section 2530 are basic components for performing distance measurement in the ToF sensor 2500. Light is emitted from the light projection section 2510, and reflected light from an object is received by the light reception section 2520. The signal processing section 2530 performs signal processing on a light reception signal to generate a distance image (depth map).

[0238] The light projection section 2510 includes a light source 2511, a collimating lens 2512, a phase modulation panel 2513, and a magnification projection lens 2514. The light source 2511 flickers to output light (pulsed light) in accordance with a light source control signal from the signal processing section 2530. The collimating lens 2512 collimates light rays from the light source 2511, and focuses the resulting parallel light rays on the phase modulation panel 2513. The phase modulation panel 2513 is reflective, and reflects the parallel light rays from the collimating lens 2512 to emit reflected light having a target intensity distribution with reference to a free form (phase distribution) provided by the free form estimation section 2550 described below. The magnification projection lens 2514 then magnifies and projects the reflected light from the phase modulation panel 2513 and emits the magnified reflected light.

[0239] The light reception section 2520 includes an image forming lens 2521 and a light reception sensor 2522. The image forming lens 2521 focuses reflected light corresponding to light emitted from the light projection section 2510 to form the reflected light as an image on a light reception surface of the light reception sensor 2522. The light reception sensor 2522 includes sensor elements arranged in an array and each generating an electric signal corresponding to an intensity of light, and outputs a light reception signal including the electric signals of the sensor elements.

[0240] The signal processing section 2530 outputs a light source control signal to the light source 2511 to cause the light projection section 251 to emit light, and performs signal processing on a light reception signal from the light reception sensor 2522 to convert a time between light projection and light reception into a distance, generating a distance image (depth map). The operation of the light projection section 2510, the light reception section 2520, and the signal processing section 2530 as described above corresponds to the operation of a typical ToF sensor.

[0241] With reference to the distance image, the target intensity distribution calculation section 2540 calculates a target intensity distribution to be formed on an entrance surface of the magnification projection lens 2514 by the phase modulation panel 2513 or in a magnification projection image on the magnification projection lens 2514 by the phase modulation panel 2513. For example, the target intensity distribution calculation section 2540 detects a region in the distance image having a reduced SN ratio of distance data due to insufficient intensity of reflected light, and calculates a target intensity distribution in a manner to obtain sufficient intensity of reflected light.

[0242] The free-form estimation section 2550 estimates a phase modulation distribution to be displayed on the phase modulation panel 2513 so that the emitted light from the phase modulation panel 2513 forms the target intensity distribution calculated by the target intensity distribution calculation section 2540. That is, the free-form estimation section 2550 estimates a free-form. In the present disclosure, the free-form estimation section 2550 estimates the free-form using a trained neural network model. The trained neural network model is mainly characterized by direct estimation of a free-form phase modulation distribution that causes a density distribution of light rays emitted from the phase modulation panel 2513 to approximate the target intensity distribution. Details of this point are as already described in items C, D, and F to H above.

[0243] [Industrial applicability]

[0244] With reference to specific embodiments, the present disclosure has been described in detail. However, it is obvious that those skilled in the art can make various modifications or substitutions to these embodiments without departing from the spirit and scope of the present disclosure.

[0245] In the present specification, embodiments in which the present disclosure is applied to an HDR-compatible projector or an image display device for a projector have been mainly described. However, the spirit and scope of the present disclosure are not limited to the HDR-compatible projector or the image display device. The present disclosure is applicable to different technologies for performing wavefront control on incident light using a spatial light modulator. For example, application of the present disclosure to various illumination devices including a light emitting section of a ToF sensor makes it possible to dynamically change an intensity distribution.

[0246] Briefly, the present disclosure has been described in an illustrative manner, and should not be construed as limiting the details set forth in this specification. The claims should be afforded the broadest reasonable interpretation so as to encompass the spirit and scope of the present disclosure.

[0247] The present disclosure can also be laid out as described below.

[0248] (1) An illumination device comprising:

[0249] a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution; and

[0250] a phase modulation section that performs phase modulation on incident light with reference to the phase modulation distribution estimated by the trained neural network model.

[0251] (2) The illumination device according to the above (1), wherein

[0252] the trained neural network model directly estimates a phase modulation distribution that achieves a light ray density distribution corresponding to the target intensity distribution.

[0253] (3) The illumination device according to the above (1) or (2), further comprising:

[0254] a luminance modulation section that performs luminance modulation on emitted light from the phase modulation section, wherein

[0255] the target intensity distribution includes a target intensity distribution calculated with reference to a target image displayed by the emitted light from the luminance modulation section, and

[0256] the luminance modulation section performs luminance modulation using a luminance modulation distribution calculated with reference to the target image and the phase modulation distribution.

[0257] (4) The illumination device according to the above (1) or (2), further comprising:

[0258] a light receiving section that receives emitted light from the phase modulation section, wherein

[0259] the target intensity distribution is set with reference to a processing result of a light receiving signal from the light receiving section.

[0260] (5) The illumination device according to the above (2), wherein

[0261] the trained neural network model is trained with reference to learning data including a set of intensity distributions input to the neural network model and phase modulation distributions used as training data.

[0262] (6) The illumination device according to the above (2), wherein

[0263] The neural network model is trained in an unsupervised manner based on an error between an intensity distribution calculated from a phase modulation distribution estimated by the neural network model and an intensity distribution input to the neural network model.

[0264] (7) The illumination device according to the above (2), wherein

[0265] The neural network model is trained in an unsupervised manner based on an error between an inverse of an intensity distribution input to the neural network model and an inverse of an intensity distribution calculated from a phase modulation distribution estimated by the neural network model with reference to a ray optics model.

[0266] (8) The illumination device according to the above (1), wherein

[0267] The trained neural network model estimates a layout of light rays corresponding to the target intensity distribution, and

[0268] The illumination device further includes a calculation section that calculates a phase modulation distribution with reference to the layout of light rays.

[0269] (9) The illumination device according to the above (8), wherein

[0270] The calculation section calculates the phase modulation distribution from the layout of light rays using an algorithm for reconstructing a surface from a gradient field.

[0271] (10) The illumination device according to the above (8), wherein

[0272] The trained neural network model is trained with reference to learning data including an intensity distribution input to the neural network model and a set of layouts of light rays used as training data.

[0273] (11) The illumination device according to the above (8), wherein

[0274] The trained neural network model is trained in an unsupervised manner based on an error between an intensity distribution calculated from a layout of light rays estimated by the neural network model and an intensity distribution input to the neural network model with reference to a ray optics model.

[0275] (12) The illumination device according to the above (8), wherein

[0276] The trained neural network model is trained in an unsupervised manner based on an error between an inverse of an intensity distribution input to the neural network model and an inverse of an intensity distribution calculated from a layout of light rays estimated by the neural network model with reference to a ray optics model.

[0277] (13) A method for generating a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution, the method comprising:

[0278] an input step of inputting an intensity distribution to the neural network model;

[0279] an evaluation step of evaluating a phase modulation distribution estimated from the intensity distribution by the neural network model; and

[0280] a learning step of training the neural network model with reference to a result of the evaluation.

[0281] (14) The method for generating a trained neural network model according to the above (13), in which

[0282] the evaluation step includes calculating a loss function based on an error between the phase modulation distribution estimated from the intensity distribution by the neural network model and a phase modulation distribution corresponding to the intensity distribution used as training data, and

[0283] the learning step includes training the neural network model using the loss function.

[0284] (15) The method for generating a trained neural network model according to the above (13), in which

[0285] the evaluation step includes calculating an error between the intensity distribution input to the neural network model and an intensity distribution calculated from the phase modulation distribution estimated from the intensity distribution by the neural network model with reference to a propagation calculation, and

[0286] the learning step includes training the neural network model in an unsupervised manner using the loss function based on the error.

[0287] (16) The method for generating a trained neural network model according to the above (13), in which

[0288] the evaluation step includes calculating a loss function based on an error between an inverse of the intensity distribution input to the neural network model and an inverse of an intensity distribution calculated from the phase modulation distribution estimated from the intensity distribution by the neural network model with reference to a propagation calculation, and

[0289] the learning step includes training the neural network model in an unsupervised manner using the loss function.

[0290] (17) A method for generating a trained neural network model that estimates a layout of light rays corresponding to a target intensity distribution, the method comprising:

[0291] an inputting step of inputting the intensity distribution to the neural network model;

[0292] an evaluating step of evaluating the layout of the light estimated by the neural network model from the intensity distribution; and

[0293] a learning step of training the neural network model with reference to a result of the evaluating.

[0294] (18) The method for generating a trained neural network model according to the above (17), wherein

[0295] the evaluating step includes calculating a loss function based on an error between the layout of the light estimated by the neural network model from the intensity distribution and a layout of the light used as training data corresponding to the intensity distribution, and

[0296] the learning step includes training the neural network model using the loss function.

[0297] (19) The method for generating a trained neural network model according to the above (18), further comprising:

[0298] a collecting step of collecting learning data by gradient field calculation from a phase modulation distribution to acquire a layout of the light used as training data, calculating the phase modulation distribution using a calculation algorithm for an arbitrary phase modulation distribution in a case where a certain intensity distribution is set as a target intensity distribution, and further resampling an intensity distribution calculated from the layout of the light by a reference light optical model on equidistant grid points, wherein

[0299] the inputting step includes inputting the learning data to the neural network model, and

[0300] the evaluating step includes evaluating a layout of the light estimated by the neural network model from the input learning data by comparing the layout of the light with a layout of the light used as training data.

[0301] (20) The method for generating a trained neural network model according to the above (17), wherein

[0302] the evaluating step includes calculating a loss function based on an error between the intensity distribution input to the neural network model and an intensity distribution calculated from the layout of the light estimated by the neural network model from the intensity distribution by a reference light optical model, and

[0303] the learning step includes training the neural network model using the loss function.

[0304] (21) The method for generating a trained neural network model according to the above (17), in which

[0305] The evaluation step includes calculating a loss function based on an error between an inverse of the intensity distribution input to the neural network model and an inverse of an intensity distribution calculated from the layout of the light rays estimated by the neural network model from the intensity distribution with reference to a ray optics model, and

[0306] The learning step includes training the neural network model using the loss function.

[0307] (22) The method for generating a trained neural network model according to the above (20) or (21), in which

[0308] The learning step includes training the neural network model using a loss function including a regularization term representing an average value of magnitudes of components of a rotation field of the layout of the light rays.

[0309] (23) A computer program described in a computer-readable format for executing a process on a computer to generate a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution, the computer program causing the computer to function as:

[0310] an input section that inputs an intensity distribution to a neural network model;

[0311] an evaluation section that evaluates a phase modulation distribution estimated by the neural network model from the intensity distribution; and

[0312] a learning section that trains the neural network model with reference to a result of the evaluation.

[0313] (24) A computer program described in a computer-readable format for executing a process on a computer to generate a trained neural network model that estimates a phase modulation distribution corresponding to a target intensity distribution, the computer program causing the computer to function as:

[0314] an input section that inputs an intensity distribution to a neural network model;

[0315] an evaluation section that evaluates a layout of light rays estimated by the neural network model from the intensity distribution; and

[0316] a learning section that trains the neural network model with reference to a result of the evaluation.

[0317] [List of Reference Numerals]

[0318] 100 projector

[0319] 101 phase modulation panel

[0320] 102 brightness modulation panel

[0321] 103 amplification optical system

[0322] 104 screen

[0323] 201 target intensity distribution calculation section

[0324] 202 freeform estimation section

[0325] 203 brightness modulation distribution calculation section

[0326] 2400 illumination device

[0327] 2401 phase modulation panel

[0328] 2402 control section

[0329] 2500 ToF sensor

[0330] 2510 light projection section

[0331] 2511 light source

[0332] 2512 collimator lens

[0333] 2513 phase modulation panel

[0334] 2514 amplification projection lens

[0335] 2520 light reception section

[0336] 2521 image formation lens

[0337] 2522 light reception sensor

[0338] 2530 signal processing section

[0339] 2540 target intensity distribution calculation section

[0340] 2550 freeform estimation section

Claims

1. A lighting device, comprising: The trained neural network model estimates the phase modulation distribution corresponding to the target intensity distribution; as well as The phase modulation unit performs phase modulation on the incident light with reference to the phase modulation distribution estimated by the trained neural network model. The trained neural network model is configured to directly estimate the phase modulation distribution from the target intensity distribution. The trained neural network model directly estimates the phase modulation distribution of the light density distribution corresponding to the target intensity distribution.

2. The lighting device according to claim 1, further comprising: The brightness modulation unit performs brightness modulation on the emitted light from the phase modulation unit, wherein... The target intensity distribution includes a target intensity distribution calculated with reference to a target image displayed by light emitted from the brightness modulation unit, and The luminance modulation unit performs luminance modulation using a luminance modulation distribution calculated with reference to the target image and the phase modulation distribution.

3. The lighting device according to claim 1, further comprising: The light receiving unit receives the emitted light from the phase modulation unit, wherein, The target intensity distribution is set with reference to the processing results of the optical received signal of the optical receiver.

4. The lighting device according to claim 1, wherein, The trained neural network model is trained using reference learning data, which includes a set consisting of the intensity distribution of the input neural network model and the phase modulation distribution used as training data.

5. The lighting device according to claim 1, wherein, The neural network model is trained in an unsupervised manner based on the error between the intensity distribution of the input neural network model and the intensity distribution calculated from the phase modulation distribution estimated from the neural network model using reference propagation.

6. The lighting device according to claim 1, wherein, The neural network model is trained in an unsupervised manner based on the error between the inverse of the intensity distribution input to the neural network model and the inverse of the intensity distribution calculated from the phase modulation distribution estimated from the neural network model using reference propagation.

7. The lighting device according to claim 1, wherein, The trained neural network model estimates the layout of light rays corresponding to the target intensity distribution, and The lighting device further includes a computing unit that calculates the phase modulation distribution with reference to the layout of the light rays.

8. The lighting device according to claim 7, wherein, The computation unit uses an algorithm for reconstructing surfaces from gradient fields to calculate the phase modulation distribution from the layout of the light rays.

9. The lighting device according to claim 7, wherein, The trained neural network model is trained using reference learning data, which includes a set consisting of the intensity distribution input to the neural network model and the layout of the rays used as training data.

10. The lighting device according to claim 7, wherein, The trained neural network model is trained in an unsupervised manner based on the error between the intensity distribution input to the neural network model and the intensity distribution calculated by the reference ray optical model from the layout of the rays estimated by the neural network model.

11. The lighting device according to claim 7, wherein, The trained neural network model is trained in an unsupervised manner based on the error between the inverse of the intensity distribution input to the neural network model and the inverse of the intensity distribution calculated by the reference ray optical model from the layout of the rays estimated by the neural network model.

12. A method for generating a trained neural network model, said trained neural network model estimating a phase modulation distribution corresponding to a target intensity distribution, said method comprising: The input step involves inputting the intensity distribution into the neural network model; The evaluation step evaluates the phase modulation distribution estimated from the intensity distribution using the neural network model. as well as The learning steps involve training the neural network model based on the evaluation results. The trained neural network model is configured to directly estimate the phase modulation distribution from the target intensity distribution. The trained neural network model directly estimates the phase modulation distribution of the light density distribution corresponding to the target intensity distribution.

13. The method for generating a trained neural network model according to claim 12, wherein, The evaluation step includes calculating a loss function based on the error between the phase modulation distribution estimated from the intensity distribution by the neural network model and the phase modulation distribution corresponding to the intensity distribution and used as training data, and The learning steps include using the loss function to train the neural network model.

14. The method for generating a trained neural network model according to claim 12, wherein, The evaluation step includes calculating the error between the intensity distribution input to the neural network model and the intensity distribution calculated from the phase modulation distribution estimated by the neural network model based on the intensity distribution, and... The learning steps include training the neural network model in an unsupervised manner using a loss function based on the error.

15. The method for generating a trained neural network model according to claim 12, wherein, The evaluation step includes calculating a loss function for the error between the reciprocal of the intensity distribution based on the input of the neural network model and the reciprocal of the intensity distribution calculated by reference propagation from the phase modulation distribution estimated by the neural network model from the intensity distribution, and The learning steps include training the neural network model in an unsupervised manner using the loss function.

16. A method for generating a trained neural network model, the trained neural network model estimating the layout of light rays corresponding to a target intensity distribution, the method comprising: The input step involves inputting the intensity distribution into the neural network model; The evaluation step evaluates the layout of the light rays estimated by the neural network model based on the intensity distribution. as well as The learning steps involve training the neural network model based on the evaluation results. The trained neural network model is configured to directly estimate the layout of the light rays from the target intensity distribution.

17. The method for generating a trained neural network model according to claim 16, wherein, The evaluation step includes calculating a loss function based on the error between the layout of the light rays estimated from the intensity distribution by the neural network model and the layout of the light rays used as training data corresponding to the intensity distribution, and The learning steps include using the loss function to train the neural network model.

18. The method for generating a trained neural network model according to claim 17, further comprising: The collection step involves collecting learning data by obtaining the ray layout used as training data from the phase modulation distribution through gradient field calculation; calculating the phase modulation distribution using a calculation algorithm for arbitrary phase modulation distributions when a specific intensity distribution is set as the target intensity distribution; and further resampling the intensity distribution calculated from the ray layout using the reference ray optical model at equally spaced grid points. The input step includes inputting the learning data into the neural network model, and The evaluation step includes evaluating the layout of the light rays estimated by the neural network model based on the input learning data by comparing the layout of the light rays with the layout of the light rays used as the training data.

19. The method for generating a trained neural network model according to claim 16, wherein, The evaluation step includes calculating a loss function for the error between the intensity distribution based on the input of the neural network model and the intensity distribution calculated by the reference ray optical model from the intensity distribution based on the layout of the rays estimated from the intensity distribution, and... The learning steps include using the loss function to train the neural network model.

20. The method for generating a trained neural network model according to claim 16, wherein, The evaluation step includes calculating a loss function for the error between the inverse of the intensity distribution based on the input neural network model and the inverse of the intensity distribution calculated by a reference ray optical model based on the layout of the rays estimated from the intensity distribution via the neural network model, and The learning steps include using the loss function to train the neural network model.

21. The method for generating a trained neural network model according to claim 19, wherein, The learning step includes training the neural network model using a loss function, the loss function including a regularization term representing the average magnitude of the components of the rotational field of the layout of the light rays.

22. A computer-readable storage medium storing a computer program described in a computer-readable format for performing processing on a computer to generate a trained neural network model, said trained neural network model estimating a phase modulation distribution corresponding to a target intensity distribution, said computer program causing the computer to function as: The input section inputs the intensity distribution into the neural network model; The evaluation unit evaluates the phase modulation distribution estimated from the intensity distribution by the neural network model; and The learning department trains the neural network model based on the evaluation results. in, The trained neural network model is configured to directly estimate the phase modulation distribution from the target intensity distribution. The trained neural network model directly estimates the phase modulation distribution of the light density distribution corresponding to the target intensity distribution.

23. A computer-readable storage medium storing a computer program described in a computer-readable format for performing processing on a computer to generate a trained neural network model, the trained neural network model estimating the layout of light rays corresponding to a target intensity distribution, the computer program causing the computer to function as: The input section inputs the intensity distribution into the neural network model; The evaluation department evaluates the layout of the light rays estimated from the intensity distribution using the neural network model. as well as The learning department trains the neural network model based on the evaluation results. The trained neural network model is configured to directly estimate the layout of the light rays from the intensity distribution.

Images