Full-color computer-generated hologram reproduction apparatus, method, and program

Abstract

To provide a full-color computer-generated hologram reproduction device, method, and program with which it is possible to adjust a color balance, taking into account the color balance of CGH images.SOLUTION: A full-color computer-generated hologram reproduction device 1 is constituted mainly by a color balance comparison unit 10 and a reproduction light illumination unit 20. The color balance comparison unit 10 includes a rendering unit 101 and a multiplier calculation unit 102 for adjustment, and, accepting as input a 3D model that is the source of a CGH reproduction image and a reference image that can, in effect, simulate the colors of the reproduction image, calculates multipliers (AR, AG, AB) for adjustment on the basis of the respective color balances of the rendering image and the reference image. The reproduction light illumination unit 20 adjusts the RGB intensity ratio of a light source using the multipliers (AR, AG, AB) for adjustment, and illuminates the CGH using reproduction light which has had its RGB intensity adjusted, so as to reproduce the hologram in color.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus, method, and program for adjusting the color balance of a full-color reconstructed image of a computer-generated hologram (CGH). [Background technology]

[0002] Because CGH uses the diffraction of light to reconstruct the CGH image (hereinafter sometimes referred to as the CGH reconstructed image or simply the reconstructed image), the interference fringes calculated differ for each wavelength. Therefore, in colorizing CGH, it is necessary to calculate the RGB interference fringes separately and superimpose the respective reconstructed images.

[0003] Patent Document 1 discloses a method for color reproduction in which interference fringes calculated using wavelengths corresponding to multiple color channels such as RGB are spatially divided and multiplexed to integrate into interference fringes on a single plate, color filters of the colors corresponding to each interference fringe region are attached to the surface, white light is used to illuminate it, and the reproduced images of each color are synthesized. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2017-219824 [Non-patent literature]

[0005] [Non-Patent Document 1] Kenji Kinoshita, "Color Image Simulation of Fresnel CGH," ITE Technical Report 29 (48), 9-12, 2005-09 [Non-Patent Document 2] Tatsuya Kobayashi, Haruhisa Kato, Akio Yoneyama, "Investigation into improving the accuracy of 2D-3D matching in the detection of the posture of three-dimensional objects", 2012 Annual Meeting of the Institute of Image Information and Television Engineers [Non-Patent Document 3] Yasuki Yamauchi, "Measurement of Individual Visual Sensitivity," Optical Society of Japan, Optics, 34(6) 311-313, June 2005. [Overview of the project] [Problems that the invention aims to solve]

[0006] Patent Document 1 spatially divides interference fringes for multiple colors, which are calculated separately without considering the intensity ratio of each color. Therefore, the RGB color balance cannot be maintained during the calculation or production process of the interference fringes.

[0007] Therefore, a technical challenge arose where, if a color model was input in which the intensity ratios of each color were not balanced, such as when the RGB intensity ratio was (R,G,B)=(8:1:1) with a particular color being stronger than the others, the color balance of the reproduced image would deviate from the intended color balance, resulting in a loss of quality.

[0008] The object of the present invention is to solve the above technical problems and to provide a full-color computer-generated hologram reproduction apparatus, method, and program that can adjust the color balance of the CGH-reproduced image to the intended original color balance. [Means for solving the problem]

[0009] To achieve the above objective, the present invention is characterized by comprising the following configuration in a full-color computer-generated hologram reproduction device that reproduces a 3D model CGH in full color.

[0010] (1) The system comprises means for calculating an adjustment multiplier for adjusting the color balance of the reproduced image based on a comparison of the colors of the rendered image of the 3D model with the colors of a reference image that mimics the CGH reproduced image, and means for adjusting the intensity ratio of at least RGB of the reproduced light based on the adjustment multiplier.

[0011] (2) A means is provided to generate a simulated image of the reconstructed image using CGH data, and this simulated image is used as a reference image to calculate the adjustment multiplier.

[0012] (3) The system further includes means for capturing a reconstructed image with reconstructed light in which the intensity ratio of at least RGB is adjusted based on an adjustment multiplier, and means for determining the completion of the adjustment of the intensity ratio of at least RGB. The means for calculating the adjustment multiplier calculates the adjustment multiplier using the captured image of the reconstructed image as a reference image, and the means for determining the completion of the adjustment determines that the adjustment is complete when a predetermined completion condition is met between the color of the rendering image and the color of the captured image. The system repeats capturing the reconstructed image, calculating the adjustment multiplier using the captured image, and adjusting the intensity ratio of at least RGB until the predetermined completion condition is met.

[0013] (4) A means for generating a simulation image of the reconstructed image using CGH data is further provided, and the means for calculating the adjustment multiplier first calculates an initial value of the adjustment multiplier using the simulation image as a reference image, then calculates the adjustment multiplier using an image of the reconstructed image captured by the reconstructed light in which at least the RGB intensity ratio has been adjusted based on the initial value as a reference image, and thereafter repeats the calculation of the adjustment multiplier using the captured image as a reference image.

[0014] (5) The means for capturing the reconstructed image is a camera equipped with an image sensor, and means for correcting the adjustment multiplier based on the relationship between the wavelength sensitivity characteristics of the image sensor and the color matching function are provided.

[0015] (6) The CHG is constructed by spatially multiplexing interference fringe regions corresponding to at least R, G, and B colors, and a color filter of the corresponding color is placed on the surface of each interference fringe region. As the regenerated light, a white structured illumination image, in which the image region corresponding to each interference fringe region is gray-cased at a density according to the adjustment multiplier, is illuminated through the color filter.

[0016] Furthermore, the present invention can be realized not only as a full-color computer-generated hologram reproduction device equipped with the above-described configurations, but also as a full-color computer-generated hologram reproduction method in which the processing of each configuration is a procedure, or as a full-color computer-generated hologram reproduction program that causes a computer to execute such procedure. [Effects of the Invention]

[0017] (1) The color balance of the rendered image of the 3D model displayed as a CGH playback image is considered to be the intended original color balance, and the RGB light source intensity is adjusted so that the colors of the reference image match the colors of the rendered image. This makes it possible to match the color balance of the playback image to the intended original color balance and improve its quality.

[0018] (2) By using a simulation image generated using CGH data as a reference image, it is possible to simulate the reconstructed image without taking a reconstructed image, and it also eliminates the need for processing to correct the difference between the color of the captured image and the color that a person perceives visually.

[0019] (3) By using the captured image of the reconstructed image as a reference image, the color balance of the reconstructed image actually observed by the observer can be matched to the color balance of the rendered image. Furthermore, since the capture of the reconstructed image, the calculation of adjustment multipliers based on the captured image, and the adjustment of the RGB intensity ratio are repeated until a predetermined termination condition is met, the quality of the reconstructed image can be improved to the desired quality.

[0020] (4) By using both the simulation image and the captured image as reference images, the initial value of the adjustment multiplier is calculated using the simulation image as the reference image first, and thereafter the adjustment multiplier is calculated using the captured image as the reference image, and this process is repeated until a predetermined termination condition is met. This allows the quality of the reproduced image to be improved to the desired quality with a small number of adjustments.

[0021] (5) By correcting the adjustment multiplier based on the relationship between the wavelength sensitivity characteristics of the image sensor that converts the reconstructed image into an electrical signal and the color matching functions of vision, the color of the captured image can be matched to the color that a person perceives, thereby improving the accuracy when adjusting the RGB light source intensity using the captured image of the reconstructed image as a reference image.

[0022] (6) As the regenerated light, a structured illumination image in which the image region corresponding to each interference fringe region is gray-cased with an intensity according to the adjustment multiplier is illuminated through a color filter. Therefore, even in colorization methods that use color filters, it is possible to adjust the RGB intensity ratio while suppressing the deterioration of the color balance due to the reflection of light from the color filter. [Brief explanation of the drawing]

[0023] [Figure 1] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to the first embodiment of the present invention. [Figure 2] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to a second embodiment of the present invention. [Figure 3] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to a third embodiment of the present invention. [Figure 4] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to the fourth embodiment of the present invention. [Figure 5] This is a flowchart illustrating the operation of the fourth embodiment. [Figure 6] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to a fifth embodiment of the present invention. [Figure 7] This is a flowchart illustrating the operation of the fifth embodiment. [Figure 8] This is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device according to the sixth embodiment of the present invention. [Figure 9] This figure illustrates a method for regenerating CGH using structured illumination images as regeneration light. [Figure 10] This figure shows an example of structured lighting images. [Modes for carrying out the invention]

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of main parts of a full-color computer-generated hologram reproducing apparatus 1 according to a first embodiment of the present invention, which mainly includes a color balance comparison unit 10 and a reproduction light illumination unit 20.

[0025] Such a full-color computer-generated hologram reproducing apparatus 1 can be configured by installing an application (program) that implements each function described in detail below on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, and the like. Alternatively, it can also be configured as a dedicated machine or a single-function machine in which a part of the application is implemented as hardware or software.

[0026] The color balance comparison unit 10 includes a rendering unit 101 and an adjustment multiplier calculation unit 102. The color balance comparison unit 10 receives, as inputs, a 3D model that is a source of a CGH reproduced image and a reference image that can substantially simulate the color of the reproduced image, and calculates adjustment multipliers (A R ,A G ,A B ) based on the respective color balances of the rendered image and the reference image. This adjustment multiplier is a vector value for adjusting the RGB intensity ratio of the reproduction light by multiplying the current RGB intensity ratio (I R ,I G ,I B ) managed by the reproduction light illumination unit 20.

[0027] The reproduction light illumination unit 20 includes a light source (not shown) with an adjustable RGB intensity ratio and an intensity ratio adjustment unit 201. The reproduction light illumination unit 20 adjusts the RGB intensity ratio of the light source using the adjustment multipliers (A R ,A G ,A B ) to perform color reproduction of the hologram by illuminating the CGH with reproduction light whose RGB intensity ratio has been adjusted.

[0028] In the color balance comparison unit 10, the rendering unit 101 renders the 3D model to generate a rendered image. Commonly used rendering techniques such as the z-buffer method or ray tracing method can be used to render the 3D model.

[0029] The adjustment multiplier calculation unit 102 calculates the average value (M) of the color histogram of the rendered image of the 3D model at a predetermined viewpoint. R M G M B ) and the average value of the color histogram of the reference image from the same viewpoint (M ' R M ' G M ' B ) and the trade (M R / M ' R M G / M ' G M B / M ' B The normalized vector of ) is adjusted by the adjustment multiplier (A R ,A G ,A B ) is calculated as follows. Alternatively, the adjustment multiplier may be calculated using other representative values ​​such as the median, maximum, or mode of the color histogram instead of the mean.

[0030] The regenerative light illumination unit 20 has a current RGB intensity ratio (I R ,I G ,I B ) with the above adjustment multiplier (A R ,A G ,A B The RGB intensity ratio of the color light source is adjusted by multiplying by (A), and the adjusted RGB intensity ratio (A R ×I R ,A G ×I G ,A B ×I B The regenerated light is illuminated using a normalized RGB intensity ratio.

[0031] According to this embodiment, the color balance of the rendered image of the 3D model displayed as a CGH regenerated image is considered to be the intended original color balance, and the RGB light source intensity is adjusted so that the colors of the reference image match the colors of the rendered image. This makes it possible to match the color balance of the regenerated image to the intended original color balance, thereby improving its quality.

[0032] In the above embodiment, color CGH reproduction using a light source with three primary colors, RGB, was described as an example. However, the present invention is not limited to this, and can be similarly applied to color reproduction using at least three primary colors, RGB, such as color CGH reproduction using a light source with four primary colors, RGBY, which includes yellow Y.

[0033] Figure 2 is a functional block diagram showing the configuration of the main parts of the full-color computer-generated hologram reproduction device 1 according to the second embodiment of the present invention, where the same reference numerals represent the same or equivalent parts. This embodiment is characterized by the use of a simulated image of CGH data as a reference image that can simulate the color of the CGH reproduced image, and the 3D model and CGH data are input to the color balance comparison unit 10.

[0034] The color balance comparison unit 10 comprises a simulation image generation unit 103, which simulates CGH data to generate a simulation image of the CGH reconstructed image. The method for generating the simulation image is disclosed in Non-Patent Literature 1, for example, but is not limited to that method. The simulation image generation unit 103 calculates the RGB intensity ratio (I) of the reconstructed light. R ,I G ,I B ) is the current RGB intensity ratio (I R ,I G ,I B The simulation image is generated assuming it is identical to ).

[0035] The adjustment multiplier calculation unit 102 uses the generated simulation image as a reference image that can simulate the CHG reconstructed image, and calculates an adjustment multiplier (A) based on the comparison result between its color histogram and the color histogram of the rendered image of the 3D model. R ,A G ,A B ) is calculated. In this embodiment as well, it is desirable to compare the color histograms of the simulation image and the rendered image, which are viewed from the same viewpoint as the CGH reconstructed image.

[0036] The intensity ratio adjustment unit 201 of the regenerative light illumination unit 20 adjusts the current RGB intensity ratio (I R ,I G ,I B ) with the above adjustment multiplier (A R ,A G ,A B The RGB intensity ratio of the color light source is adjusted by multiplying by (A), and the adjusted RGB intensity ratio (A R ×I R ,A G ×I G ,A B ×I B The CGH is illuminated with regenerated light of a normalized RGB intensity ratio.

[0037] According to this embodiment, since a simulation image generated using CGH data is used as a reference image, it is possible to simulate the reconstructed image without taking a reconstructed image, and furthermore, processing to correct the difference between the color of the captured image and the color that a person perceives visually is unnecessary.

[0038] Figure 3 is a functional block diagram showing the configuration of the main parts of the full-color computer-generated hologram reproduction device 1 according to the third embodiment of the present invention, where the same reference numerals represent the same or equivalent parts. This embodiment is characterized in that it employs a captured image of the hologram reproduction as a reference image that can simulate the color balance of the CHG reproduction image, and the 3D model and the captured image are input to the color balance comparison unit 10.

[0039] The full-color computer-generated hologram reproduction device 1 mainly consists of a color balance comparison unit 10, a reproduction light illumination unit 20, and a reproduction image capture unit 30. The color balance comparison unit 10 further comprises a completion determination unit 104 and a camera pose estimation unit 105. The reproduction image capture unit 30 can be a camera equipped with an image sensor or a spectrometer, but here we will explain using the case where a camera is used as an example.

[0040] The reconstructed image capture unit 30 captures the CGH reconstructed image from a predetermined viewpoint and feeds the captured image back to the color balance comparison unit 10. The color balance comparison unit 10 compares the color histogram of the rendered image of the 3D model with the color histogram of the captured image and adjusts the adjustment multiplier (A R ,A G ,A B Calculate ).

[0041] In this embodiment as well, it is desirable to compare the color histograms of the captured image and the rendered image of the CGH reconstructed image viewed from the same viewpoint. The camera pose estimation unit 105 calculates the pose of the reconstructed image capture unit 30 using a 3D-2D matching method with the captured image, and estimates the viewpoint based on the calculation result. The rendering unit 101 generates a rendered image of the 3D model from the estimated viewpoint.

[0042] The regenerative light illumination unit 20 has a current RGB intensity ratio (I R ,I G ,I B ) with the above adjustment multiplier (A R ,A G ,A B The RGB intensity ratio of the regenerated light is adjusted by multiplying by (A), and the adjusted RGB intensity ratio (A R ×I R ,A G ×I G ,A B ×I B The CGH is illuminated with regenerated light of a normalized RGB intensity ratio. The steps of calculating the adjustment multiplier, adjusting the RGB intensity ratio, and providing feedback on the captured image are repeated until the termination determination unit 104 determines that the process is complete.

[0043] The completion determination unit 104 determines that, for example, (1) the number of adjustments by the adjustment multiplier A R , A G , A B ) has reached a predetermined upper limit, (2) the difference between each color histogram of the rendered image and the captured image is less than a predetermined threshold Th R , Th G , Th B ), or (3) it can be visually determined that the color balance is sufficiently matched, and determines the end of adjustment of the RGB intensity ratio with at least one of these as the termination condition.

[0044] If "(1) the number of adjustments of the adjustment multiplier" is employed as the termination condition, the final adjustment multiplier A R , A G , A B ) may be the value obtained when the number of adjustments reaches the upper limit, or may be the value obtained when the difference in color histogram shows the minimum value in a plurality of illumination intensity ratio adjustments before reaching the upper limit.

[0045] If "(2) difference in color histograms" is employed as the termination condition, the average value of the color histogram of the captured image used for calculating the adjustment multiplier M' R , M' G , M' B ) and the average value of the color histogram of the rendered image M R , M G , M B ) can be compared with the threshold Th R , Th G , Th B ).

[0046] Note that, instead of the difference between the average values of the color histograms, a difference between other representative values such as a median value or a mode value may be used for the termination determination. Furthermore, since this difference is not intended to correct the ratio of each color unlike the adjustment multiplier, the difference may be compared using an evaluation index that compares the overall tendency, such as the correlation between color histograms, cosine similarity, or Earth mover's distance, instead of using a representative value.

[0047] In the termination determination based on the difference, the difference of the color histogram for each color channel is compared with each corresponding threshold value Th R , Th G , Th B ), and when all differences fall below the respective threshold values, that is, |M' R -M R |<Th R , |M' G -M G |<Th G , |M' B -M B |<Th B termination may be determined when all of the above conditions are satisfied at the same time.

[0048] On the other hand, when the color balance is extremely deviated, for example, one color channel is hardly included in the color of a 3D model, there are cases where only the color channel having little influence on the appearance does not satisfy the termination condition, and the adjustment of the illumination intensity cannot be completed. In this case, if determination is performed based on the magnitude of a vector as shown in the following formula (1) instead of performing determination for each color channel, robust termination determination can be achieved even when a specific channel does not converge.

[0049] [Formula]

[0050] Furthermore, these two termination conditions may be combined, and the determination method may be switched according to the deviation of the color histogram: when the deviation is estimated to be equal to or greater than a predetermined value from the representative values (M R , M G , M B ) of the color histogram of the rendered image, determination based on the magnitude of the vector is performed, otherwise determination is performed for each color channel.

[0051] Furthermore, similar to the termination condition in (1), the differences between each color histogram may be compared using evaluation metrics that compare the overall trend, such as correlation between color histograms, cosine similarity, and Earth mover's distance. In addition, the adjustment multiplier (A R ,A G ,A B The initial value of ) can be set to a predetermined value (for example, (1 / 3, 1 / 3, 1 / 3)) for the first adjustment, and subsequent adjustments can be made based on the captured image.

[0052] According to this embodiment, since the captured image of the reconstructed image is used as a reference image, the color balance of the reconstructed image actually observed by the observer can be matched to the color balance of the rendered image. Furthermore, since the capture of the reconstructed image, the calculation of adjustment multipliers based on the captured image, and the adjustment of the RGB intensity ratio are repeated until a predetermined termination condition is met, the quality of the reconstructed image can be improved to the desired quality.

[0053] Figure 4 is a functional block diagram showing the configuration of the main parts of the full-color computer-generated hologram reproduction device 1 according to the fourth embodiment of the present invention, and is a configuration that combines the second and third embodiments.

[0054] This embodiment is characterized in that it first compares the color histogram of the rendered 3D model image with the color histogram of the simulation image to calculate an adjustment multiplier, and then compares it with the color histogram of the captured image to calculate an adjustment multiplier, repeating this process until the termination determination unit 104 determines termination.

[0055] Figure 5 is a flowchart showing the operation of this embodiment. In step S1, the adjustment multiplier calculation unit 102 of the color balance comparison unit 10 calculates an initial value of the adjustment multiplier (A) based on the comparison result between the color histogram of the rendered image and the color histogram of the simulation image. R0 ,A G0 ,A B0 ) is calculated.

[0056] In step S2, the intensity ratio adjustment unit 201 of the regenerative light illumination unit 20 adjusts the current RGB intensity ratio (I R ,I G ,I B ) to the aforementioned initial value (A R0 ,A G0 ,A B0 The RGB intensity ratio is adjusted by multiplying by ). In step S3, the regenerated light with the adjusted RGB intensity ratio is illuminated to the CGH and the regenerated image of the CGH is reproduced. In step S4, the regenerated image is captured by the regenerated image capture unit 30 and the captured image is fed back to the color balance comparison unit 10.

[0057] In step S5, the adjustment multiplier calculation unit 102 calculates an adjustment multiplier (A) based on the comparison result between the color histogram of the rendered image and the color histogram of the captured image. R ,A G ,A B The value is recalculated. The rendered image is regenerated by rendering the 3D model at the viewpoint estimated by the camera pose estimation unit 105 using the captured image.

[0058] In step S6, the termination determination unit 104 of the color balance comparison unit 10 performs a termination determination. If the termination determination is not successful, the process returns to step S2, and the current RGB intensity ratio (I R ,I G ,I B ) with the above adjustment multiplier (A R ,A G ,A B The RGB intensity ratio is readjusted by multiplying by ). In step S3, the regenerated light with the readjusted RGB intensity ratio is used to illuminate the CGH again.

[0059] In response to this, if the termination condition is met, the process will be terminated, and the current RGB intensity ratio (I R ,I G ,I B ) is the final RGB intensity ratio (I R ,I G ,I B ) will be maintained as such.

[0060] According to this embodiment, a simulation image and a captured image are used as reference images, and initially, the simulation image is used as the reference image to calculate the initial value of the adjustment multiplier. Thereafter, the captured image is used as the reference image to calculate the adjustment multiplier, and this process is repeated until a predetermined termination condition is met. As a result, the quality of the reproduced image can be improved to the desired quality with a small number of adjustments.

[0061] Figure 6 is a functional block diagram showing the configuration of the main parts of the full-color computer-generated hologram reproduction apparatus 1 according to the fifth embodiment of the present invention, where the same reference numerals represent the same or equivalent parts. This embodiment is characterized by the inclusion of an illumination intensity ratio correction unit 40 in addition to the color balance comparison unit 10, the reproduction light illumination unit 20, and the reproduction image capture unit 30.

[0062] The illumination intensity ratio correction unit 40 comprises at least the first correction unit 401 of the first correction unit 401 and the second correction unit 402, and an adjustment multiplier (A R ,A G ,A B After the color balance adjustment is complete, a correction multiplier (B) is used to correct the RGB intensity, taking into account the wavelength sensitivity characteristics of the camera and the human eye. R ,B G ,B B ) is calculated. Then, the correction multiplier (B R ,B G ,B B ) adjusts multiplier (A R ,A G ,A B Multiply by ) to get the final adjusting multiplier (A R ,A G ,A B Calculate ).

[0063] The intensity ratio adjustment unit 201 of the regenerated light illumination unit 20 adjusts the RGB intensity ratio (I) using the final adjustment multiplier output by the illumination intensity ratio correction unit 40. R ,I G ,I B The RGB intensity ratio correction unit 40 corrects the following two points.

[0064] The first objective is to correct the difference between the color of the captured image of the reconstructed image and the color that a person perceives visually. This is because, due to the wavelength sensitivity characteristics of the image sensor mounted on the camera that captures the reconstructed image, even if the color balance of the reconstructed image and the simulation image match, a person may perceive them as not matching. This correction is handled by the first correction unit 401.

[0065] The second objective is to compensate for individual differences in color perception. This is because there are slight individual differences in how the human eye perceives colors, and if we consider the use of CGH for content where, for example, differences in color tone within the reproduced image have a significant impact on perceived quality, it is desirable to individually optimize the color tone. This correction is handled by the second correction unit 402.

[0066] The color of an image captured by a camera of a hologram's reconstructed image is obtained by converting the tristimulus value, calculated by multiplying the wavelength distribution R(λ) of the reconstructed image's light by the wavelength sensitivity characteristic S(λ) of the image sensor, into the RGB color space. When a person observes the reconstructed image with their eyes, it is thought that they perceive the color by replacing the camera's wavelength sensitivity characteristic S(λ) with a color matching function H(λ).

[0067] Here, the first correction coefficient (B') is used to achieve the first objective. 1R ,B' 1G ,B' 1B ) can be calculated as the ratio of the wavelength sensitivity characteristic S(λ) and the color matching function H(λ). Therefore, if the average value of the wavelength sensitivity characteristic S(λ) is given by equation (2) below and the average value of the color matching function H(λ) is given by equation (3) below, the first correction coefficient (B' 1R ,B' 1G ,B' 1B ) can be calculated using the following formula (4).

[0068]

number

[0069] The second correction coefficient (B') to achieve the second objective 2R ,B' 2G ,B'2B ) is not the standard observer's color matching function H(λ), but rather the unique color matching function H measured for each individual i. i It can be calculated using (λ). For example, the method disclosed in Non-Patent Document 2 can be used to measure the color matching function Hi(λ) for each individual i.

[0070] In this embodiment, the illumination intensity ratio correction unit 40 is used to correct the wavelength sensitivity characteristic S(λ) and the color matching function H(λ), H i The first correction unit 401 holds (λ) and applies the wavelength sensitivity characteristic S(λ) and the color matching function H(λ) to the above equations (2)-(4) to obtain the first correction multiplier (B' R1 ,B' G1 ,B' B1 The second correction unit 401 calculates the color matching function H of individual i. i The ratio of (λ) to the standard observer's color matching function H(λ) is given by the first correction multiplier (B' R1 ,B' G1 ,B' B1 By multiplying by ), the second correction multiplier (B' 2R ,B' 2G ,B' 2B Calculate ).

[0071] Furthermore, if a spectrometer is used as the image reproduction unit 30, the light spectrum can be acquired at multiple points and converted to RGB using the color matching functions of a standard observer, such as a CIE colorimetric standard observer. In this case, the correction for the first objective described above becomes unnecessary.

[0072] Figure 7 is a flowchart showing the operation of the fifth embodiment. Since each process in steps S1 to S6 is the same as in the fourth embodiment, the explanation is omitted. Here, we will explain using the case where the color balance is corrected for individual i using the color matching function Hi(λ) of individual i as an example.

[0073] In step S6, the termination condition is met, and the RGB intensity ratio (I R ,I G ,I B Once the adjustment of ) is complete, in step S7, the first correction unit 401 calculates the first correction multiplier (B'R1 ,B' G1 ,B' B1 ) is calculated. In step S8, the second correction unit 402 multiplies the ratio of individual i's color matching function Hi(λ) to the standard observer's color matching function H(λ) by the first correction multiplier to obtain the second correction multiplier (B' R2 ,B' G2 ,B' B2 ) is calculated.

[0074] In step S9, the second correction multiplier (B' R2 ,B' G2 ,B' B2 ) is the RGB intensity ratio (I R ,I G ,I B ) is multiplied by the final adjusting multiplier (A R ,A G ,A B ) is calculated, and furthermore, the current RGB intensity ratio (I R ,I G ,I B ) to the final adjusting multiplier (A R ,A G ,A B ) is multiplied to obtain the RGB intensity ratio (I R ,I G ,I B ) is adjusted for the individual i.

[0075] According to this embodiment, by correcting the adjustment multiplier based on the relationship between the wavelength sensitivity characteristics of the image sensor that converts the reproduced image into an electrical signal and the color matching function of vision, the color tone of the captured image can be matched to the color tone perceived by humans. This further improves the accuracy when adjusting the RGB light source intensity using the captured image of the reproduced image as a reference image.

[0076] Figure 8 is a functional block diagram showing the configuration of a full-color computer-generated hologram reproduction device 1 according to the sixth embodiment of the present invention, where the same reference numerals represent the same or equivalent parts. This embodiment is characterized by the inclusion of a structured illumination image generation unit 50 in addition to the color balance comparison unit 10, the reproduction light illumination unit 20, the reproduction image capture unit 30, and the illumination intensity ratio correction unit 40.

[0077] The color balance comparison unit 10 receives the 3D model, CGH data, and captured image as input. In this embodiment, as shown in Figure 9, a color filter is attached to the surface of a display (CGH) that displays interference fringes corresponding to each RGB color channel, and the CGH reconstructed image is colorized by illuminating the CGH with white reconstructed light through the color filter.

[0078] The structured illumination image generation unit 50 utilizes the fact that interference fringes are spatially divided and multiplexed for each color channel, and generates a structured illumination image in which the brightness of each illumination range corresponding to the interference fringes of each color channel is adjusted in grayscale according to the color balance.

[0079] The luminance values ​​for the illumination range corresponding to each interference fringe of structured illumination are calculated by the color balance comparison unit 10 using an adjustment multiplier (A R ,A G ,A B Calculated based on the luminance value before adjustment (L). For example, the luminance value before adjustment (L R ,L G ,L B ) with an adjustment multiplier (A R ,A G ,A B The calculation is performed by multiplying the results and then normalizing them.

[0080] The regenerated light illumination unit 20 illuminates the CGH by displaying the structured illumination image generated by the structured illumination image generation unit 50 on a projector. The regenerated light illumination unit 20 is not limited to a projector; any device capable of illuminating structured illumination at any desired brightness can be used.

[0081] Here, because the light source has a certain spread, at the point where the luminance values ​​of a structured illumination image switch, the luminance value illuminating one color channel may affect other adjacent color channels as well.

[0082] These technical challenges can be solved by providing a margin area (light-shielding area) that does not transmit light at the point where the luminance values ​​of the structured illumination image [Figure 10(a)] switch, as shown in Figure 10(b). Alternatively, as shown in Figure 10(c), the problem can be solved by gradually changing each luminance value in a gradient-like manner at the point where the luminance values ​​switch.

[0083] According to this embodiment, as the regenerated light, a structured illumination image in which the image region corresponding to each interference fringe region is gray-cased with a density according to the adjustment multiplier is illuminated through a color filter. Therefore, even in a colorization method using a color filter, it becomes possible to adjust the color balance while suppressing the deterioration of the color balance due to light reflection by the color filter.

[0084] In each of the above embodiments, the color histogram of the rendered image is compared with the color histogram of the simulated image or captured image at a specific viewpoint position, and an adjustment multiplication (A) is performed. R ,A G ,A B This was explained as a method for calculating (A). However, the present invention is not limited to this, and the adjustment multipliers are calculated from multiple perspectives, and statistical values ​​such as their mean and median are used to calculate the final adjustment multiplier (A). R ,A G ,A B ) may be adopted as the final adjustment multiplier (A R ,A G ,A B ) is also acceptable.

[0085] Furthermore, when playing back a holographic video using an electronic device capable of switching the displayed interference fringes, adjustment multipliers may be calculated from multiple frames of the CGH-reproduced image, and the final adjustment multiplier may be calculated based on statistical values ​​such as the mean and median of these multipliers.

[0086] Furthermore, according to each of the above embodiments, the color balance of the CGH reconstructed image can be adjusted to the intended color balance, thereby improving the quality of the CGH reconstructed image. This makes it possible to contribute to the United Nations-led Sustainable Development Goal (SDG) 9, "Build resilient infrastructure and promote inclusive and sustainable industrialization," and Goal 11, "Make cities inclusive, safe, resilient and sustainable." [Explanation of Symbols]

[0087] 10...Color balance comparison unit, 20...Reconstruction light illumination unit, 30...Reconstruction image capture unit, 40...Illumination intensity ratio correction unit, 50...Structured illumination image generation unit, 101...Rendering unit, 102...Adjustment multiplier calculation unit, 103...Simulation image generation unit, 104...Termination determination unit, 105...Camera pose estimation unit, 201...Intensity ratio adjustment unit, 401...First correction unit, 402...Second correction unit

Claims

1. In a full-color computer-generated hologram playback device that reproduces a 3D model as a CGH (Computer-Generated Hologram) in full color, A means for calculating an adjustment multiplier to adjust the color balance of a reproduced image by comparing the colors of a rendered image of a 3D model with the colors of a reference image that mimics a reproduced image of a CGH, A full-color computer-generated hologram reproduction apparatus, characterized by comprising means for adjusting the intensity ratio of at least RGB of the reproduced light based on the adjustment multiplier.

2. The system includes means for generating a simulated image of the reconstructed image using the CGH data, The full-color computer-generated hologram reproduction apparatus according to claim 1, characterized in that the means for calculating the adjustment multiplier calculates the adjustment multiplier using the simulation image as a reference image.

3. A means for capturing a reconstructed image with reconstructed light in which the intensity ratio of at least RGB is adjusted based on the adjustment multiplier, The system comprises means for determining the completion of adjusting the intensity ratio of at least RGB, The means for calculating the adjustment multiplier calculates the adjustment multiplier using the captured image of the reconstructed image as a reference image, The means for determining the completion of the adjustment determines whether a predetermined completion condition is met between the color of the rendered image and the color of the captured image, The full-color computer-generated hologram reproduction apparatus according to claim 1, characterized in that it repeats the capture of the reproduced image, the calculation of adjustment multipliers based on the captured image, and the adjustment of at least the RGB intensity ratio until the predetermined termination condition is met.

4. The system further comprises means for generating a simulated image of the reconstructed image using the CGH data, The means for calculating the adjustment multiplier is: First, the initial value of the adjustment multiplier is calculated using the simulation image as the reference image. Next, an adjustment multiplier is calculated using the reference image as the image of the regenerated image captured by the regenerated light, in which at least the RGB intensity ratio has been adjusted based on the initial value. The full-color computer-generated hologram reproduction apparatus according to claim 3, characterized in that it repeats the capture of the reproduced image, the calculation of adjustment multipliers based on the captured image, and the adjustment of at least the RGB intensity ratio until the predetermined termination condition is met.

5. The means for capturing the reconstructed image is a camera equipped with an image sensor. The full-color computer-generated hologram reproduction apparatus according to claim 3, further comprising means for correcting an adjustment multiplier based on the relationship between the wavelength sensitivity characteristics of the image sensor and the color matching function.

6. The means for capturing the reconstructed image is a camera equipped with an image sensor. The full-color computer-generated hologram reproduction apparatus according to claim 4, further comprising means for correcting an adjustment multiplier based on the relationship between the wavelength sensitivity characteristics of the image sensor and the color matching function.

7. The aforementioned CHG is constructed by spatially multiplexing interference fringe regions corresponding to at least each of the R, G, and B colors. A color filter of the corresponding color is placed on the surface of each interference fringe region. The full-color computer-generated hologram reproduction apparatus according to claim 1, characterized in that the regeneration light is a structured illumination image in which the image region corresponding to each interference fringe region is gray-cased at a density corresponding to the adjustment multiplier, illuminated through a color filter.

8. The full-color computer-generated hologram reproduction apparatus according to any one of claims 1 to 7, characterized in that the means for calculating the adjustment multiplier calculates the adjustment multiplier based on the ratio of the representative value of the color histogram of the rendering image to the representative value of the color histogram of the reference image.

9. The full-color computer-generated hologram reproduction apparatus according to claim 8, characterized in that the representative value is any of the mean, maximum, minimum, and mode.

10. The means for calculating the adjustment multiplier is characterized by calculating the adjustment multiplier by comparing the color of the rendered image of the reconstructed image at the same viewpoint with the color of the reference image, as described in any one of claims 1 to 7.

11. The full-color computer-generated hologram reproduction apparatus according to any one of claims 1 to 7, characterized in that the means for calculating the adjustment multiplier is to compare each color of the rendered image and the reference image from multiple viewpoints to calculate the adjustment multiplier.

12. The full-color computer-generated hologram playback apparatus according to any one of claims 1 to 7, characterized in that the reproduced image is a holographic video, and the final adjustment multiplier is calculated based on statistical values ​​of adjustment multipliers calculated from multiple frames of the reproduced image.

13. The system includes means for estimating the camera's viewpoint based on the aforementioned captured image, The full-color computer-generated hologram playback apparatus according to any one of claims 3 to 6, characterized in that it compares the color of the rendered image of the 3D model from the estimated viewpoint with the color of the captured image.

14. The means for determining the completion of the adjustment is to obtain the difference between representative values ​​for at least RGB of each color histogram of the rendered image and the captured image for each color channel, compare it with a predetermined first threshold, and determine that the adjustment is complete if the difference for at least one color channel is less than the corresponding first threshold, as described in any one of claims 3 to 6.

15. The means for determining the completion of the adjustment is to compare the difference between the vectors of representative values ​​for at least each RGB of the color histograms of the rendered image and the captured image with a predetermined second threshold, and if the difference is less than the second threshold, it is determined that the adjustment is complete, as described in any one of claims 3 to 6.

16. The means for determining the completion of the adjustment estimates the bias of the color histogram of the rendered image, If the bias is less than a predetermined value, the difference between the representative values ​​of at least RGB in the color histograms of the rendered image and the captured image is calculated for each color channel and compared with a predetermined first threshold. If the difference for at least one color channel is less than the corresponding first threshold, the adjustment is determined to be complete. The full-color computer-generated hologram reproduction apparatus according to any one of claims 3 to 6, characterized in that if the bias is greater than or equal to a predetermined value, the difference between the vectors of representative values ​​for at least each RGB of the color histograms of the rendered image and the captured image is compared with a predetermined second threshold, and if the difference is less than the second threshold, it is determined that the adjustment is complete.

17. A means for estimating the viewpoint from which the reconstructed image was taken based on the captured image, The full-color computer-generated hologram playback apparatus according to any one of claims 3 to 6, characterized in that the rendered image is an image of the 3D model viewed from that viewpoint.

18. The full-color computer-generated hologram reproduction apparatus according to any one of claims 3 to 6, characterized in that the means for capturing the reconstructed image is a spectrometer.

19. The means for correcting the adjustment multiplier comprises means for calculating a first correction multiplier based on the ratio of the wavelength sensitivity characteristics of the image sensor to a standard color matching function, The full-color computer-generated hologram reproduction apparatus according to claim 5 or 6, characterized in that the adjustment multiplier, for which the adjustment of at least the RGB intensity ratio has been completed, is corrected by the first correction multiplier.

20. The means for correcting the adjustment multiplier comprises means for calculating a second correction multiplier based on the ratio of the standard color matching function to the observer's color matching function, The full-color computer-generated hologram reproduction apparatus according to claim 19, characterized in that the adjustment multiplier, for which the adjustment of at least the RGB intensity ratio has been completed, is corrected by the first and second correction multipliers.

21. The full-color computer-generated hologram reproduction apparatus according to claim 7, characterized in that a light-shielding region is provided at the boundary of the interference fringe region corresponding to each color of the structured illumination image.

22. The full-color computer-generated hologram reproduction apparatus according to claim 7, characterized in that the density changes in steps at the boundaries of the interference fringe regions corresponding to each color of the structured illumination image.

23. A full-color computer-generated hologram reproduction method in which a computer reproduces a CGH of a 3D model in full color, By comparing the colors of the rendered image of the 3D model with the colors of a reference image that mimics the CGH-reproduced image, an adjustment multiplier is calculated to adjust the color balance of the reproduced image. A method for reproducing a full-color computer-generated hologram, characterized by adjusting the intensity ratio of at least RGB of the regenerated light based on the adjustment multiplier.

24. A full-color computer-generated hologram playback program for playing back a CGH of a 3D model in full color, A procedure for calculating an adjustment multiplier to adjust the color balance of a reproduced image by comparing the colors of a rendered image of a 3D model with the colors of a reference image that mimics the reproduced image of a CGH, and A full-color computer-generated hologram reproduction program characterized by causing a computer to perform a procedure for adjusting the intensity ratio of at least RGB of the reproduced light based on the adjustment multiplier.

Images