Computer-generated hologram reconstructing device and its structured illumination calibration method and program

Abstract

To provide a device, method and program which can highly accurately calibrate a structured illumination image of animation CGH.SOLUTION: A camera 10 images an interference fringe region in which interference fringes are spatially multiplexed on a hologram surface 60 and an irradiation region of a structured illumination image irradiated to the hologram surface 60 by a projector 50. An observation image input unit 20 acquires images of the interference fringe region and the irradiation region of the structured illumination image captured by the camera 10 and temporarily stores them. An irradiation region decision unit 30 recognizes the position of the interference fringe region in a pixel coordinate system on the projection side on the basis of a camera image of the interference fringe region and decides the position of the interference fringe region as a target irradiation region of the structured illumination image. An irradiation adjustment unit 40 discriminates the irradiation region of the structured illumination image on the hologram surface 60 on the basis of the camera image and adjusts (calibrates) a projection condition of the structured illumination image with the pixel coordinate system such that the structured illumination image is mapped on the target irradiation region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer-generated hologram reconstruction device and a structured illumination calibration method and program thereof, and more particularly to a computer-generated hologram reconstruction device and a structured illumination calibration method and program thereof that are suitable for reconstructing animation CGHs. [Background technology]

[0002] Among computer-generated holograms (CGHs), a full-parallax high-definition CGH (FPHD-CGH) is known, which uses a laser direct imaging device to print fine interference fringes on a pattern drawing material, allowing a high-resolution reconstructed image to be viewed from any viewpoint.

[0003] A method has been proposed for FPHD-CGH that allows animation playback by switching between multiple frames. In this method, interference fringes from multiple frames (frame 1, frame 2) are first extracted as shown in Figure 14, and then integrated into a single interference fringe region by space division multiplexing.

[0004] Next, to irradiate only the interference fringes corresponding to one frame with structured illumination light, the projector projects an image (structured illumination image) in which the illuminated area is white and the other areas are black, as shown in Figure 15. An animation CGH is realized by sequentially switching and playing back the interference fringes of the illuminated frames.

[0005] It is known that animation CGHs can have problems such as not being able to accurately irradiate the interference fringes of the desired frame with structured illumination light, resulting in the desired frame not being reproduced in some areas of the reconstructed image, or multiple frames being reproduced simultaneously, causing crosstalk. For this reason, animation CGHs generally require accurate adjustment (calibration) of the illumination area of ​​the structured illumination image to match the interference fringe area. Here, the illumination area of ​​the structured illumination image refers to the position (coordinates of the four corners) where the projector irradiates the structured illumination image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-102728 Summary of the Invention [Problem to be solved by the invention]

[0007] Currently, no technology has been proposed for automatically and accurately calibrating the illumination area of ​​a structured illumination image of an animation CGH. Therefore, the illumination area of ​​a structured illumination image must generally be manually adjusted by the user while checking the reconstructed image of the animation CGH, and highly accurate calibration of the illumination area of ​​a structured illumination image is time-consuming.

[0008] An object of the present invention is to solve the above technical problems and provide a computer-generated hologram reconstruction device that can calibrate the illumination area of ​​a structured illumination image of an animation CGH with high accuracy in a short time, as well as a structured illumination calibration method and program for the same. [Means for solving the problem]

[0009] In order to achieve the above object, the present invention provides a computer-generated hologram reconstruction device that reconstructs an animation CGH by illuminating a structured illumination image onto an interference fringe region where interference fringes for multiple frames are spatially multiplexed, and is characterized by having the following configuration.

[0010] (1) The system includes a means for acquiring camera images of an interference fringe region in which interference fringes are spatially multiplexed and an illumination region of structured illumination light, a means for determining the position of the interference fringe region in a target illumination region of the structured illumination image based on the camera images, and a means for adjusting the illumination region of the structured illumination image to the target illumination region based on the camera images.

[0011] (2) The system is equipped with a means for performing whole-frame illumination to uniformly illuminate all interference fringes and obtain a whole-frame illumination image, a means for performing single-frame illumination to selectively illuminate one frame's interference fringe on multiple illumination areas to obtain multiple single-frame illumination images, a means for calculating an illumination area score representing the amount of crosstalk in each single-frame illumination image based on the difference between the whole-frame illumination image and each single-frame illumination image, and a means for adjusting the illumination area of ​​the structured illumination image based on the illumination area score of each single-frame illumination image.

[0012] (3) The device includes a means for performing reduced illumination in an illumination area of ​​the adjusted structured illumination image by reducing the illumination range corresponding to each interference fringe of the single-frame illumination to acquire a reduced illumination image, and a means for calculating an illumination range score representing the amount of crosstalk of each reduced illumination image based on a change in the total pixel value of each reduced illumination image acquired by repeatedly reducing the illumination range, and the adjusting means adjusts the illumination range for each interference fringe of the structured illumination image based on the illumination range score.

[0013] The present invention can be realized not only as a computer-generated hologram reconstruction device having such a characteristic configuration, but also as a structured illumination calibration method for a computer-generated hologram reconstruction device that uses such characteristic processing as a procedure, or as a structured illumination calibration program for a computer-generated hologram reconstruction device that causes a computer to execute such a procedure. [Effects of the Invention]

[0014] According to the present invention, the following effects can be achieved.

[0015] (1) It will be possible to automatically calibrate the illuminated area of ​​structured illumination images in animated CGHs quickly and accurately.

[0016] (2) Calibration is performed based on the amount of crosstalk that affects the viewer's subjective quality, enabling calibration that directly contributes to improving subjective quality.

[0017] Furthermore, even if a camera with a resolution sufficient to capture a reconstructed image of an animation CGH is used, calibration of structured illumination light with little crosstalk can be achieved.

[0018] Furthermore, the amount of crosstalk can be estimated from the change in the reconstructed image of the animation CGH when the irradiation area of ​​the structured illumination light and the irradiation range for each interference fringe are changed, so the effect of calibration errors on the reconstructed image can be estimated in advance.

[0019] Furthermore, since the calibration is performed while observing the changes in the reconstructed image of the animation CGH, the calibration of the structured illumination light can be realized without the need for target points.

[0020] (3) Since it is possible to calibrate not only the illumination area of ​​the structured illumination image but also the illumination range corresponding to the interference fringe area of ​​each frame, it is possible to achieve local calibration of the structured illumination image in a short time with high accuracy, further improving the subjective quality of the reconstructed image. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a functional block diagram of a first embodiment of a computer-generated hologram reconstruction device to which the present invention is applied. [Figure 2] FIG. 2 is a diagram illustrating a method for calibrating a structured illumination image according to the first embodiment. [Figure 3] FIG. 10 is a functional block diagram of a second embodiment of a computer-generated hologram reconstruction device to which the present invention is applied. [Figure 4] 10A and 10B are diagrams showing an example of a full frame irradiation and a full frame irradiation image. [Figure 5] 1A and 1B are diagrams showing an example of a single-frame exposure and a single-frame exposure image. [Figure 6] 10A and 10B are diagrams showing examples in which the amount of crosstalk in each single-frame irradiation image changes depending on the relative positions of the irradiation area of ​​the single-frame irradiation and the interference fringe area. [Figure 7] 10 is a flowchart showing the operation of the second embodiment. [Figure 8] FIG. 10 is a functional block diagram of a third embodiment of a computer-generated hologram reconstruction device to which the present invention is applied. [Figure 9] 10A and 10B are diagrams illustrating an example in which crosstalk occurs in a reconstructed image due to a calibration error in an illumination region of a structured illumination image. [Figure 10] 10A and 10B are diagrams showing an example in which crosstalk in a reconstructed image is eliminated by reducing the illumination range of each interference fringe of a structured illumination image. [Figure 11] 10A and 10B are diagrams showing an example in which the illumination range score and the amount of crosstalk change in response to the reduction of the illumination range. [Figure 12] 10 is a flowchart showing the operation of the third embodiment. [Figure 13]FIG. 11 is a diagram showing an example in which a structured illumination image is spatially divided into a plurality of parts and an illumination range is determined for each divided part in the third embodiment. [Figure 14] FIG. 10 is a diagram showing an example in which interference fringes for multiple frames are integrated into one interference fringe region by space division multiplexing. [Figure 15] FIG. 10 is a diagram for explaining a method for playing back an animation CGH. DETAILED DESCRIPTION OF THE INVENTION

[0022]

[0023] The present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 is a functional block diagram showing the configuration of a first embodiment of a computer-generated hologram reconstruction device to which the present invention is applied. The main components of the device include a camera 10, an observation image input unit 20, an illumination region determination unit 30, an illumination adjustment unit 40, and a projector 50.

[0023] Such a computer-generated hologram reconstruction device can be configured by installing applications (programs) that realize the functions detailed below on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated or single-function machine in which part of the application is implemented as hardware or software.

[0024] Camera 10 captures an interference fringe region on hologram surface 60 where interference fringes are spatially multiplexed, and an area illuminated by structured illumination light projected onto hologram surface 60 by projector 50. In this embodiment, when capturing an image of the area illuminated by structured illumination light, projector 50 is caused to irradiate only the four corners of the area illuminated, and camera 10 captures an image of the entire hologram surface 60. Observation image input unit 20 acquires and temporarily stores images of the interference fringe region and the area illuminated by structured illumination light captured by camera 10.

[0025] The illumination area determination unit 30 recognizes the position of the interference fringe area in the pixel coordinate system on the projection side based on the camera image of the interference fringe area, and determines the position of the interference fringe area as the target illumination area of ​​the structured illumination image. Note that it is also possible to capture an image of a marker or target point whose relative position with respect to the interference fringe area is known, and recognize the position of the interference fringe area based on the marker or target point.

[0026] The illumination adjustment unit 40 determines the illumination area of ​​the structured illumination image on the hologram surface 60 based on a camera image including the four corner points of the illumination area, and adjusts (calibrates) the projection conditions of the structured illumination image in a pixel coordinate system so that the structured illumination image is mapped to the target illumination area, as shown in an example in Fig. 2. In this embodiment, the illumination adjustment unit 40 includes a projection transformation unit 401, which adjusts the illumination area by projectively transforming the current illumination area of ​​the structured illumination image based on its relative positional relationship with the target illumination area.

[0027] The four corner points of the current illumination area of ​​the structured illumination image are defined as (p i ,q i ) (i=1~4), the four corner points of the target illumination area of ​​the structured illumination image are (r i ,s i ) (i=1 to 4), the relative positional relationship between these points can be expressed by the homography matrix H of the following equation (1). Here, u i is the left side of [p i q i 1] T is a coefficient to make the third component of 1.

[0028]

number

[0029] The homography matrix H in the above equation (1) is i ,q i ) and point (r i ,s i) can be calculated using the least squares method from a set of four corresponding points. Using this homography matrix H, the projective transformation unit 401 associates each pixel position in the current illumination area of ​​the structured illumination image with each pixel position in the target illumination area. The projector 50 projects the structured illumination image with the adjusted illumination area onto the hologram surface 60 to generate a reconstructed image 70.

[0030] The adjustment of the illumination area of ​​the structured illumination image by the illumination adjustment unit 40 may be repeated until a predetermined termination condition is satisfied. That is, the homography matrix H may be recalculated based on the correspondence between the adjusted illumination area of ​​the structured illumination image and the target illumination area, and the structured illumination image obtained by projectively transforming the illumination area using the recalculated homography matrix H may be irradiated, and this process may be repeated until the predetermined termination condition is satisfied.

[0031] The termination condition may be defined as the number of iterations of the adjustment, or may be that the distance between the four corresponding points falls below a predetermined value. According to this embodiment, it is possible to realize high-speed and high-precision automatic calibration of structured illumination images in animation CGHs.

[0032] However, the above-described first embodiment still has the following four technical problems. First, in order to recognize the position of the interference fringe region using camera 10 and calibrate the irradiation region of the structured illumination image, a camera with extremely high resolution that can recognize interference fringes is required. Therefore, it is difficult to accurately calibrate the irradiation region of the structured illumination light using a general camera.

[0033] Second, it is difficult to estimate in advance the effect that calibration errors will have on the reconstructed image, making it difficult to perform calibration that directly contributes to improving subjective quality.

[0034] Third, when calibration is performed using markers or target points instead of recognizing interference fringes, the calibration accuracy may be reduced due to misalignment of the installed markers or target points or an insufficient number of them.

[0035] Fourth, since calibration of the illumination area involves calibrating the illumination position of the entire structured illumination image, there is a possibility that the calibration accuracy will be reduced locally, for example, by irradiating the structured illumination image onto interference fringes of frames that are not desired to be reproduced on a part of the hologram surface.

[0036] Therefore, in the second and third embodiments of the present invention described in detail below, the amount of crosstalk is estimated from the reconstructed image of the animation CGH captured by the camera 10, thereby estimating the effect of calibration errors on the reconstructed image, and high-precision calibration that can directly contribute to improving subjective quality is realized without requiring a high-resolution camera or marker target points.

[0037] 3 is a functional block diagram showing the configuration of a second embodiment of a computer-generated hologram reconstruction device to which the present invention is applied, and the same reference numerals as those used above represent the same or equivalent parts. This embodiment is characterized in that irradiation area determination unit 30 includes all-frame irradiation image acquisition unit 301, single-frame irradiation image acquisition unit 302, image storage unit 303, irradiation area score calculation unit 304, and irradiation area score storage unit 305.

[0038] As shown in an example in Figure 4, the all-frame irradiation image acquisition unit 301 performs all-frame irradiation [Figure 4(a)], in which structured illumination light is uniformly irradiated onto the interference fringes of all frames, and acquires the reconstructed image observed during all-frame irradiation (hereinafter referred to as all-frame irradiation image) [Figure 4(b)], which is then stored in the image storage unit 303. The brightness of all-frame irradiation is preferably half the brightness when each frame is irradiated with structured illumination light and reconstructed, but the brightness may be the same. With all-frame irradiation, all frames are reconstructed simultaneously, which maximizes the amount of crosstalk that occurs.

[0039] 5, the single-frame irradiation image acquisition unit 302 performs single-frame irradiation [Fig. 5(a)], which selectively irradiates structured illumination light only onto interference fringes corresponding to any single frame, and acquires a reconstructed image (hereinafter referred to as a single-frame irradiation image) [Fig. 5(b)] observed during single-frame irradiation and stores it in the image storage unit 303. Single-frame irradiation is repeated while randomly selecting multiple positions that are candidate irradiation areas, and a single-frame irradiation image is stored for each irradiation area.

[0040] The irradiation area score calculation unit 304 calculates an irradiation area score representing the amount of crosstalk of each single-frame irradiation image based on the difference between the full-frame irradiation image and each single-frame irradiation image, and stores the calculation result in the irradiation area score storage unit 305. In this embodiment, a vector in which each pixel value of the full-frame irradiation image is arranged in a row is defined as y∈R n (n is the number of pixels in the observed image, R n is a set of n-dimensional real vectors), and the vector in which each pixel value of a single exposure image is arranged in a row is x∈R n The irradiation area score is calculated using the following equation (2):

[0041]

number

[0042] FIG. 6 is a diagram showing an example in which the amount of crosstalk in each single-frame irradiation image changes depending on the relative position between the irradiation area of ​​the single-frame irradiation and the interference fringe area.

[0043] The amount of crosstalk decreases when the irradiation area of ​​the single-frame irradiation is changed so that the mean square error between each pixel value of the single-frame irradiation image and each pixel value of the full-frame irradiation image increases. Therefore, the amount of crosstalk can be reduced by adjusting the irradiation area of ​​the single-frame irradiation so that the irradiation area score increases.

[0044] FIG. 7 is a flowchart showing the operation of the second embodiment, and mainly shows the operation of the irradiation region determining unit 30.

[0045] In step S101, all-frame illumination is performed, in which structured illumination light is uniformly applied to the interference fringes of all frames. In step S102, all-frame illumination images reproduced by the all-frame illumination are captured by camera 10 and stored in image storage unit 303.

[0046] In step S103, an illumination area for single-frame illumination is randomly selected, in which structured illumination light is irradiated only onto the interference fringes corresponding to an arbitrary single frame. In step S104, single-frame illumination is performed on the randomly selected illumination area. In step S105, a single-frame illumination image reproduced by the single-frame illumination is captured by camera 10 and stored in image storage unit 303.

[0047] In step S106, the irradiation area score for estimating the amount of crosstalk between the all-frame irradiation image stored in the image storage unit 303 and the current single-frame irradiation image is calculated using the above formula (2). In step S107, the irradiation area score is stored in association with the irradiation area of ​​the current single-frame irradiation.

[0048] In step S108, it is determined whether a predetermined termination condition is met. If not, the process returns to step S103, where another irradiation area is randomly selected and the process of calculating the irradiation area score is repeated.

[0049] The termination condition can be defined as the number of times the crosstalk amount is estimated by repeating single-frame irradiation or the irradiation area score exceeding a predetermined threshold, or alternatively, it may be defined as at least one of the number of single-frame irradiations and the irradiation area score exceeding a threshold.

[0050] When the termination condition is satisfied, the process proceeds to step S109, where the illumination area for single-frame illumination corresponding to the illumination area score with the smallest amount of crosstalk (the largest illumination area score) among the illumination area scores stored in step S107 is determined as the illumination area of ​​the structured illumination image. The illumination adjustment unit 40 adjusts the illumination area of ​​the structured illumination image to the determined illumination area.

[0051] According to this embodiment, calibration is performed based on the amount of crosstalk that affects the subjective quality of the viewer, so that calibration that can directly contribute to improving the subjective quality becomes possible.

[0052] Furthermore, according to this embodiment, the camera 10 is not required to have a high resolution, and any camera that can capture a reproduced image of an animation CGH with sufficient resolution can realize calibration of the structured illumination image.

[0053] Generally, observing extremely fine interference fringes with a pixel pitch of about 1 μm requires a camera with a resolution several thousand times higher than 4K. Meanwhile, due to the characteristics of human vision, the fovea can be considered an observation system with a resolution of about 10 million pixels, meaning that a resolution of, for example, 8K (approximately 33 million pixels) can be adequately simulated. In this embodiment, calibration is performed based on the subjective quality of the reproduced image actually viewed by the user from changes in the reproduced image, so crosstalk can be reduced by using a camera with a resolution of 8K or higher.

[0054] Furthermore, according to this embodiment, the amount of crosstalk can be estimated from the change in the reconstructed image of the animation CGH when the irradiation area of ​​the structured illumination light is changed, so the effect of calibration errors on the reconstructed image can be estimated in advance.

[0055] Furthermore, according to this embodiment, calibration is performed while observing changes in the reconstructed image of the animation CGH, so that calibration of structured illumination light can be achieved without the need for target points.

[0056] In addition, although the present embodiment has been described as selecting the candidate irradiation areas randomly, rules may be created for the method of selecting the irradiation areas. For example, a selection range of the candidate irradiation areas may be defined, and the four corners of the irradiation area may be changed within the selection range according to raster scanning, zigzag scanning, or Morton code. In this case, the amount of change in the four corners may be set to be smaller depending on the irradiation area score, for example, as the irradiation area score increases.

[0057] 8 is a functional block diagram showing the configuration of a third embodiment of a computer-generated hologram reconstruction device to which the present invention is applied, and the same reference numerals as those used above denote the same or equivalent parts. This embodiment is characterized in that the illumination area determination unit 30 further includes a reduced illumination image acquisition unit 306, an illumination area score calculation unit 307, and an illumination area score accumulation unit 308.

[0058] In the second embodiment, if the position of the illumination area of ​​the adjusted structured illumination image is shifted to the right due to a calibration error, for example, as shown in FIG. 9(a), and crosses over into the interference fringe area of ​​another frame, crosstalk will occur in the reconstructed image, as shown in FIG. 9(b).

[0059] Therefore, in this embodiment, while maintaining the position of the illumination area of ​​the structured illumination image, as shown in FIG. 10(a), only the illumination range of the structured illumination light corresponding to each interference fringe of the structured illumination image is reduced, thereby preventing the occurrence of crosstalk as shown in FIG. 10(b).

[0060] The reduced-irradiation image acquisition unit 306 performs reduced-irradiation in which the irradiation range of the structured illumination light corresponding to each interference fringe is reduced by a predetermined ratio in the irradiation area of ​​single-frame irradiation determined based on the amount of crosstalk in the second embodiment, and acquires a reconstructed image (hereinafter referred to as a reduced-irradiation image) observed during reduced-irradiation and stores it in the image storage unit 303. The reduced-irradiation is repeated while reducing the irradiation range by a predetermined ratio, and a reduced-irradiation image is stored for each reduced irradiation range.

[0061] The irradiation area score calculation unit 307 calculates an irradiation area score that can represent the amount of crosstalk of each reduced irradiation image based on the change in the total pixel value of the reduced irradiation image obtained each time the irradiation area is reduced by a predetermined ratio. The irradiation area score of each reduced irradiation image is stored in the irradiation area score storage unit 308.

[0062] As shown in Figure 11, if the irradiation range, which includes not only the first frame of the irradiation target but also the interference fringes of the adjacent second frame, is reduced and the total pixel value of the reduced irradiation image is calculated each time, the total pixel value of the reduced irradiation image gradually decreases as the reduction proceeds.

[0063] However, at a certain point, the structured illumination light is no longer irradiated onto the interference fringes of the second frame, and the crosstalk disappears. After that, although the structured illumination light is irradiated onto only the interference fringes of the first frame, the irradiated area gradually shrinks, and the total pixel value gradually decreases.

[0064] Therefore, it can be assumed that the reduction in the total pixel value of the reduced irradiation image when crosstalk occurs is significantly different from the reduction in the total pixel value of the reduced irradiation image after crosstalk no longer occurs. In other words, the reduction rate of the total pixel value relative to the reduction rate of the irradiation range changes specifically at the crosstalk vanishing point.

[0065] Therefore, in this embodiment, the reduction irradiation is repeated and attention is paid to the three most recent consecutive reduced irradiation images, and the vector x t ∈R n ,x t-1 ∈R n ,x t-2 ∈R n is applied to the following equation (3) to calculate the illumination range score.

[0066]

number

[0067] where x t is the reduced irradiation image vector obtained in the t-th reduced irradiation, x t-1 is the reduced irradiation image vector obtained in the t-1th reduced irradiation, x t-2 represents the reduced irradiation image vector acquired in the t-2th reduced irradiation. Therefore, the size of the irradiation range in each reduced irradiation image is x t <x t-1 <x t-2 This becomes:

[0068] The above equation (3) is the second derivative of the total pixel value of the reduced-irradiation image based on the irradiation range of each interference fringe. Since the decrease in the total pixel value of the reduced-irradiation image when crosstalk occurs is different from the decrease in the total pixel value of the reduced-irradiation image after crosstalk no longer occurs, the irradiation range score at the time when crosstalk no longer occurs is greater than the irradiation range score calculated at other times. In the above equation (3), the time of the t-1th reduced-irradiation is the vanishing point of crosstalk.

[0069] Therefore, in this embodiment, the illumination range of each interference fringe of the structured illumination light when the illumination range score has a large value is estimated as the optimal illumination range of the structured illumination light. The illumination adjustment unit 40 adjusts the illumination range of the structured illumination light corresponding to each interference fringe of the structured illumination image to the optimal illumination range.

[0070] 12 is a flowchart showing the operation of the third embodiment, and mainly shows the operation of the irradiation region determination unit 30. Note that the processing of steps S101 to S109 is the same as in the second embodiment, and therefore a description thereof will be omitted.

[0071] In step S110, in the illumination region determined in step S109, only the illumination range corresponding to each interference fringe of the structured illumination light in the single-frame illumination is reduced by a predetermined ratio. In step S111, reduced illumination is performed using the single-frame illumination image in which each illumination range has been reduced. In step S112, the reduced illumination image reproduced by the reduced illumination is photographed by camera 10 and stored in image storage unit 303.

[0072] In step S113, the irradiation range score calculation unit 307 calculates the vector x t ∈R n ,x t-1 ∈R n ,x t-2 ∈R nis applied to the above formula (3) to calculate the irradiation area score, and the calculation result is stored in the irradiation area score storage unit 308. In step S114, it is determined whether a predetermined termination condition is met. If not, the process returns to step S110, and the above processes are repeated while further reducing the irradiation area.

[0073] The termination condition can be defined as the number of times the reduced irradiation is repeated or the irradiation range score for which the crosstalk amount is estimated exceeding a predetermined threshold, or alternatively, it may be defined as at least one of the number of times the reduced irradiation is repeated and the irradiation range score exceeding a predetermined threshold.

[0074] On the other hand, if the predetermined termination condition is met, the process proceeds to step S115, where the vector x t-1 The illumination range corresponding to the vanishing point of the crosstalk is determined as the illumination range corresponding to the vanishing point of the crosstalk. The illumination adjustment unit 40 adjusts the illumination range of the structured illumination image to the determined illumination range.

[0075] In this embodiment, all the irradiation range scores stored in the irradiation range score storage unit 308 are referenced, and the vector x t-1 The illumination range corresponding to the crosstalk vanishing point is determined as the illumination range corresponding to the crosstalk vanishing point. If there is no illumination range score exceeding a predetermined threshold, the illumination range may not be reduced. If there are multiple illumination range scores exceeding the threshold, the smallest illumination range among the illumination ranges corresponding to the illumination range scores may be determined.

[0076] In the above embodiment, the calibration of the illumination range corresponding to each interference fringe is performed after the calibration of the illumination area of ​​the structured illumination image is completed. However, the present invention is not limited to this, and each calibration process may be performed alternately.

[0077] According to this embodiment, it is possible to calibrate not only the illumination area of ​​the structured illumination image but also the illumination range of the structured illumination light corresponding to the interference fringes of each frame of the structured illumination image, thereby achieving high-precision calibration of the structured illumination image even locally, and further improving the subjective quality of the reproduced image.

[0078] Furthermore, according to this embodiment, the amount of crosstalk can be estimated from the change in the reconstructed image of the animation CGH when the irradiation range of the structured illumination light corresponding to each interference fringe is changed, so that the effect of calibration errors on the reconstructed image can be estimated in advance.

[0079] In this embodiment, the irradiation range score is calculated using the above formula (3), but other lp parameters may be calculated using the following formula (4).

[0080]

number

[0081] Furthermore, in the present embodiment, the illumination range of the structured illumination light for each interference fringe is uniformly reduced. However, the structured illumination image may be spatially divided into a plurality of portions, and the illumination range may be adaptively determined for each divided portion.

[0082] FIG. 13 is a diagram showing a schematic diagram of a method for determining the illumination range for each part of a structured illumination image, and shows an example in which the structured illumination image is divided into two parts vertically and horizontally, for a total of four parts.

[0083] First, in the determined illumination area [Fig. 1(a)], the illumination range (white portion) is gradually reduced by focusing on the upper left portion [Fig. 1(b)] of the structured illumination image, and an illumination range score is calculated from the reconstructed image that can be observed at that time. The illumination range for the upper left portion is then determined based on the illumination range score. This series of processes is then repeated, switching the focus to the upper right portion [Fig. 1(c)], lower left portion [Fig. 1(d)], and lower right portion [Fig. 1(e)] in sequence, to determine the illumination range for each portion.

[0084] Furthermore, the illumination range score may be calculated for each color channel of the observed image. In this case, when the illumination range score for any color channel has a large value, the corresponding illumination range may be taken as the calibration result.

[0085] In addition, in this embodiment, the illumination range of each interference fringe of the structured illumination light is described as being reduced from the right end, but it may also be reduced from the left end, top end, or bottom end, or it may be reduced by a combination of these, or it may be reduced according to any rule.

[0086] Although the above embodiments have been described using an animation CGH of two frames as an example, the present invention is not limited to this and can be similarly applied to an animation CGH of three or more frames.

[0087] Furthermore, according to each of the above embodiments, it is possible to automate the calibration of structured illumination images of animation CGHs, thereby improving the subjective quality of the reproduced images, which makes it possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0088] 10... camera, 20... observation image input unit, 30... irradiation area determination unit, 40... irradiation adjustment unit, 50... projector, 60... hologram surface, 70... reconstructed image, 301... whole frame irradiation image acquisition unit, 302... single frame irradiation image acquisition unit, 303... image storage unit, 304... irradiation area score calculation unit, 305... irradiation area score storage unit, 306... reduced irradiation image acquisition unit, 307... irradiation area score calculation unit, 308... irradiation area score storage unit, 401... projection transformation unit

Claims

1. A computer-generated hologram reproducing device that reproduces an animation CGH (Computer-Generated Hologram) by irradiating a structured illumination image onto an interference fringe region where interference fringes for multiple frames are spatially multiplexed, a means for performing full frame irradiation to uniformly irradiate all interference fringes and acquire a full frame irradiation image; a means for selectively irradiating a plurality of irradiation areas with single-frame irradiation to obtain a plurality of single-frame irradiation images; a means for calculating an exposure area score representing the amount of crosstalk of each single-frame exposure image based on a difference between the full-frame exposure image and each single-frame exposure image; and means for adjusting the illumination area of ​​the structured illumination image based on the illumination area score of each single-frame illumination image.

2. a means for determining a target illumination area of ​​the structured illumination image based on the illumination area score of each single-frame illumination image; 2. The computer-generated hologram reconstructing apparatus according to claim 1, wherein said adjusting means adjusts the illumination area of ​​the structured illumination image to the determined target illumination area.

3. A computer-generated hologram reproduction device as described in Claim 2, characterized in that the adjusting means adjusts the illumination area of ​​the structured illumination image based on the relative positions of a plurality of reference points defining the interference fringe area and a plurality of reference points defining the target illumination area of ​​the structured illumination image.

4. 4. The computer-generated hologram reconstruction device according to claim 3, wherein the adjusting means performs projective transformation of each position within the illumination area of ​​the structured illumination image onto each position within the target illumination area using a projective transformation matrix calculated based on the relative positions.

5. 2. The computer-generated hologram reconstructing apparatus according to claim 1, wherein the plurality of irradiation areas are selected randomly.

6. a means for performing reduced illumination by reducing an illumination range corresponding to each interference fringe of the single frame illumination in an illumination area of ​​the adjusted structured illumination image, thereby acquiring a reduced illumination image; and a means for calculating an illumination range score representing the amount of crosstalk of each reduced illumination image based on a change in the total pixel value of each reduced illumination image obtained by repeatedly reducing the illumination range, 6. The computer-generated hologram reconstruction device according to claim 1, wherein the adjusting means adjusts the illumination range of each interference fringe of the structured illumination image based on the illumination range score.

7. 7. The computer-generated hologram reconstructing device according to claim 6, wherein the means for calculating the illumination range score calculates a second derivative value of the total pixel value of three consecutive reduced illumination images.

8. A method for calibrating a structured illumination image of a computer-generated hologram reproducing device that reproduces an animation CGH (Computer-Generated Hologram) by irradiating a structured illumination image onto an interference fringe region where interference fringes for multiple frames are spatially multiplexed, comprising: All frame irradiation is performed to uniformly irradiate all interference fringes, and an all frame irradiation image is obtained. A single-frame irradiation is performed on a plurality of irradiation areas to selectively irradiate the interference fringes of one frame, thereby obtaining a plurality of single-frame irradiation images; calculating an exposure area score representing the amount of crosstalk in each single-frame exposure image based on a difference between the full-frame exposure image and each single-frame exposure image; A structured illumination calibration method for a computer-generated hologram reconstructing device, comprising adjusting the illumination area of ​​a structured illumination image based on the illumination area score of each single-frame illumination image.

9. In the illumination area of ​​the adjusted structured illumination image, a reduced illumination is performed by reducing the illumination range corresponding to each interference fringe of the single frame illumination to obtain a reduced illumination image; Calculating an illumination range score representing the amount of crosstalk of each reduced illumination image based on a change in the total pixel value of each reduced illumination image obtained by repeatedly reducing the illumination range; 9. The method for calibrating structured illumination for a computer-generated hologram reconstruction device according to claim 8, further comprising adjusting an illumination range for each interference fringe of the structured illumination image based on the illumination range score.

10. A program for calibrating the structured illumination image of a computer-generated hologram reproducing device that reconstructs an animation CGH (Computer-Generated Hologram) by irradiating a structured illumination image onto an interference fringe region where interference fringes for multiple frames are spatially multiplexed. A procedure for performing full frame irradiation to uniformly irradiate all interference fringes and acquiring a full frame irradiation image; A step of performing single-frame irradiation to selectively irradiate interference fringes of one frame on a plurality of irradiation areas to obtain a plurality of single-frame irradiation images; calculating an exposure area score representing the amount of crosstalk in each single-frame exposure image based on a difference between the full-frame exposure image and each single-frame exposure image; adjusting the illumination area of ​​the structured illumination image based on the illumination area score of each single-frame illumination image; A structured illumination calibration program for a computer-generated hologram reconstructing device, characterized by causing a computer to execute the above.

11. a step of performing reduced illumination by reducing an illumination range corresponding to each interference fringe of the single-frame illumination in an illumination area of ​​the adjusted structured illumination image, thereby acquiring a reduced illumination image; and calculating an illumination range score representing the amount of crosstalk of each reduced illumination image based on a change in the total pixel value of each reduced illumination image obtained by repeatedly reducing the illumination range, 11. The structured illumination calibration program for a computer-generated hologram reconstruction device according to claim 10, wherein the adjustment step adjusts an illumination range for each interference fringe of the structured illumination image based on the illumination range score.

Images