Information processing apparatus, information processing method, program, and hologram display system

Abstract

The present invention improves user experience. The information processing apparatus includes a grouping unit (21) that groups a plurality of pixels constituting one or more objects included in one piece of image data into one or more groups, a phase adjustment unit (22) that assigns a phase pattern having a different phase difference for each of the one or more groups to the plurality of pixels, and a calculation unit (30) that generates hologram data from the image data to which the phase pattern is imparted.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, a program, and a hologram display system. BACKGROUND

[0002] A computer-generated hologram (hereinafter, also referred to as CGH) is a technique of generating a hologram only by computation. Such a technique is needed because it is often difficult to directly store a hologram of a scene.

[0003] Conventionally, a random phase is added to an input pixel or an input voxel before a wave propagation process in which information propagated from the pixel or the voxel is spread over a wide area on a hologram. When the information is spread over a wide range of the hologram plane, a reproduced image has a shallower depth of field, and robustness to speckle noise and dust attached to a lens of an optical system is improved.

[0004] However, adding a random phase to an input image causes other types of noise on a reproduced scene. Such noise is called speckle noise, and appears on a reproduced scene substantially randomly. Many techniques for reducing speckle noise are based on time-consuming iterative algorithms, which are disadvantageous in terms of real-time performance.

[0005] CGH computation algorithms that do not use a random phase have also been developed, and a reproduced image obtained by using such a method that does not use a random phase can achieve a fairly high image quality without speckle noise. In the case where a random phase is not used, a hologram having information from sampling points in space is concentrated in a narrow area. Therefore, a light beam of a reproduced image becomes thin, and thus a depth of field of the reproduced image becomes deep. In this regard, conventional techniques for mitigating these two phenomena have been developed.

[0006] LIST OF CITATIONS

[0007] PATENT LITERATURE

[0008] Patent Literature 1: JP 2005-195803 A SUMMARY

[0009] TECHNICAL PROBLEM

[0010] However, the above two phenomena always have a trade-off relationship. Therefore, in the related art, it is not possible to simultaneously reduce both the image resolution difference and the deep depth of field for the same image, and it is difficult to improve the user experience.

[0011] In this regard, the present disclosure proposes an information processing apparatus, an information processing method, and a program capable of improving the user experience.

[0012] SOLUTION TO PROBLEM

[0013] To address the above problems, an information processing apparatus according to one aspect of the present disclosure includes a grouping unit that groups a plurality of pixels included in one piece of image data, which constitute one or more objects, into one or more groups; a phase adjustment unit that assigns a phase pattern having a different phase difference for each of the one or more groups to the plurality of pixels; and a calculation unit that generates hologram data from the image data to which the phase pattern is added. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic diagram for explaining an outline of a hologram display system according to a first embodiment.

[0015] Figure 2 is a block diagram showing a schematic configuration example of an information processing apparatus according to the first embodiment.

[0016] Figure 3 is a flowchart showing an operation example of the information processing apparatus according to the first embodiment.

[0017] Figure 4 is a diagram showing an example of an object image constituting input image data according to the first embodiment.

[0018] Figure 5 is a diagram showing an example of a depth image constituting input image data according to the first embodiment.

[0019] Figure 6 is a diagram showing an example of a label image constituting input image data according to the first embodiment.

[0020] Figure 7 is a diagram showing an example of a random phase patch according to the first embodiment.

[0021] Figure 8 is a diagram showing an example of a down-converted random phase patch according to the first embodiment.

[0022] Figure 9 is a diagram showing a positional relationship between a hologram to be reproduced in the first embodiment, a hologram display system, and a viewer (user).

[0023] Figure 10 is a diagram showing an example of a hologram displayed to the user (in the case of looking at the baby) according to the first embodiment.

[0024] Figure 11 is a diagram showing an example of a hologram displayed to the user (in the case of looking at the boy behind) according to the first embodiment.

[0025] Figure 12is a block diagram showing a schematic configuration example of an information processing apparatus according to a second embodiment.

[0026] Figure 13 is a block diagram showing a schematic configuration example of an information processing apparatus according to a third embodiment.

[0027] Figure 14 is a diagram for explaining a line-of-sight direction of a user detected by a line-of-sight tracking unit according to the third embodiment.

[0028] Figure 15 is a diagram for explaining an example of grouping objects based on Figure 14 line-of-sight directions shown in FIG. 7.

[0029] Figure 16 is a diagram showing an example of a hologram displayed to a user according to the third embodiment (a case of looking at a baby in front).

[0030] Figure 17 is a diagram showing an example of a hologram displayed to a user according to the third embodiment (a case of looking at a boy behind).

[0031] Figure 18 is a block diagram showing a schematic configuration example of an information processing apparatus according to a fourth embodiment.

[0032] Figure 19 is a view showing an example of a surrounding image acquired by an imaging device according to the fourth embodiment.

[0033] Figure 20 is a diagram showing an example of a feature map generated by a feature detection unit based on a surrounding image according to the fourth embodiment.

[0034] Figure 21 is a diagram for explaining an example of grouping objects according to the fourth embodiment.

[0035] Figure 22 is a diagram showing an example of a hologram displayed to a user according to the fourth embodiment (a case of looking at a baby in front).

[0036] Figure 23 is a diagram showing an example of a hologram displayed to a user according to the fourth embodiment (a case of looking at a boy behind).

[0037] Figure 24 is a diagram for explaining a line-of-sight direction of a user detected by a line-of-sight tracking unit according to a fifth embodiment.

[0038] Figure 25 is a diagram for explaining an example of grouping objects based on Figure 24 line-of-sight directions shown in FIG. 17.

[0039] Figure 26 is a diagram showing an example of a hologram displayed to a user according to the fifth embodiment (looking at the baby case).

[0040] Figure 27 is a diagram showing an example of a hologram displayed to a user according to the fifth embodiment (looking at the boy case).

[0041] Figure 28 is a block diagram showing a hardware configuration example of an information processing apparatus according to an embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, an embodiment of the present disclosure will be described in detail based on the drawings. Incidentally, in each of the following embodiments, the same reference signs are assigned to the same portions, and overlapping description will be omitted.

[0043] The present disclosure will be described in accordance with the order of the items described below.

[0044] 1. First Embodiment

[0045] 1.1 Outline

[0046] 1.2 Schematic configuration example of information processing apparatus

[0047] 1.3 Operation example of information processing apparatus

[0048] 1.4 Effects and advantages

[0049] 2. Second Embodiment

[0050] 3. Third Embodiment

[0051] 4. Fourth Embodiment

[0052] 5. Fifth Embodiment

[0053] 6. Hardware configuration

[0054] 1. First Embodiment

[0055] Hereinafter, an information processing apparatus, an information processing method, and a program according to the first embodiment will be described in detail with reference to the drawings.

[0056] 1.1 Outline

[0057] First, an outline of the present embodiment will be described. Figure 1 is a schematic diagram for explaining an outline of a hologram display system according to the present embodiment. Note that, in Figure 1 and subsequent drawings, (A) to (F) indicate respective schematic positions.

[0058] As Figure 1 shown in the drawing, in the present embodiment, a hologram display system is shown, including a light source 101, a magnifying optical system including a plurality of lenses 102 and a lens 103, a beam splitter 104, and a spatial light modulator 105.

[0059] In Figure 1 the present embodiment, laser light L1 from the light source 101 is converted into coherent light L2 having a beam diameter that is magnified by the magnifying optical system including the plurality of lenses 102 and the lens 103. The coherent light L2 passes through the beam splitter 104 and enters the spatial light modulator 105.

[0060] In the present embodiment, the spatial light modulator 105 is shown by a reflective spatial light modulator. The coherent light L2 is modulated by the spatial light modulator 105 to form a hologram at a point in a predetermined region in space.

[0061] The beam splitter 104 projects the hologram 106 reproduced by the spatial light modulator 105, for example, at a position E in space, so that a user 107 can observe the hologram. The user 107 located at a position F can view the hologram 106 superimposed on the real space visible through the beam splitter 104 by looking in the direction of the position E.

[0062] As described above, the hologram display system according to the present embodiment can provide the following user experiences: causing a virtual object or the like to appear in a real space, rendering a subject in a real space with special effects or the like, presenting predetermined information to a user, and the like.

[0063] Note that, in the above-described configuration, the display device 50 can be, for example, an optical see-through type head-mounted display (hereinafter, referred to as an AR HMD). In addition, the pre-processing unit 20, the CGH calculation unit 30, and the post-processing unit 40 can constitute an information processing device in the hologram display system. Part or all of the information processing device can be arranged in the display device 50 (i.e., in the AR HMD), or can be arranged in a server (including a cloud server) or the like connected to the display device 50 via a predetermined network (e.g., a mobile communication network including a local area network (LAN), the Internet, Long Term Evolution (LTE), Wireless Fidelity (WiFi), 4G, 5G, or the like).

[0064] 1.2 Schematic Configuration Example of Information Processing Device

[0065] Next, a schematic configuration example of an information processing device included in the hologram display system according to the present embodiment will be described in detail with reference to the drawings. Figure 2 is a block diagram showing a schematic configuration example of an information processing device according to the present embodiment. As Figure 2As shown, the information processing apparatus 1 includes a pre-processing unit 20, a CGH calculation unit 30, and a post-processing unit 40.

[0066] (Pre-processing unit 20)

[0067] The pre-processing unit 20 includes a grouping unit 21 and a phase adjustment unit 22, and performs pre-processing described later on input image data (input image data 10 described later).

[0068] (CGH calculation unit 30)

[0069] The CGH calculation unit 30 generates hologram data to be input on an SLM plane by calculation from input image data pre-processed by the pre-processing unit 20.

[0070] (Post-processing unit 40)

[0071] The post-processing unit 40 converts the hologram data generated by the CGH calculation unit 30 into a hologram signal displayable on the display apparatus 50.

[0072] (Display apparatus 50)

[0073] The display apparatus 50 outputs the hologram signal converted by the post-processing unit 40 to stereoscopically display a hologram of an object reproduced from the input image data to a user.

[0074] 1.3 Operation example of information processing apparatus

[0075] Next, an operation example of the information processing apparatus 1 according to the present embodiment will be described in detail with reference to the drawings. Figure 3 is a flowchart showing an operation example of the information processing apparatus according to the present embodiment.

[0076] As Figure 3 shown, in the present operation, first, in step S101, the information processing apparatus 1 inputs information (hereinafter, referred to as input image data) related to an image to be displayed as a hologram. In the present embodiment, the input image data 10 can be, for example, 2-dimensional image data, 2.5-dimensional image data, 3-dimensional image data, or the like. Note that the 2.5-dimensional image data can be, for example, image data including color information of three primary colors of RGB and depth information (also referred to as distance information) for each pixel or voxel (hereinafter, simply referred to as pixel). In addition, the three-dimensional image data can be, for example, image data including color information of three primary colors of RGB and three-dimensional information.

[0077] In the following description, a case where 2.5-dimensional image data is input as the input image data 10 will be described as an example. Figure 4 and Figure 5 is an example of input image data according to the present embodiment,Figure 4 An image data (referred to as an object image) including RGB color information of each pixel in the input image data 10 is shown, Figure 5 An image data including depth information of each pixel (hereinafter, referred to as a depth image) is shown.

[0078] Figure 4 The shown object image G40 includes an object C41 of a boy located at the upper left, an object C42 of a baby located at the lower right, and an object C43 of a clock located at the lower left. In the following description, in the case where a distinction is made between an object to be displayed to a subject in a real space (in this example, the objects C41 and C42) and an object to be displayed on a screen virtually set in a virtual setting (in this example, the object C43), the object to be displayed to a subject in a real space is referred to as a virtual object (including a performance effect, etc.) (hereinafter, referred to as an AR object), and the object to be displayed on a screen virtually set on a predetermined plane in a real space is referred to as an OSD object. Note that, in the following description, the object to be displayed to a subject in a real space is referred to as an AR object, and the object to be displayed on a screen virtually set in a virtual setting is referred to as an OSD object. Figure 4 In the shown object image G40, the white background region R41 indicates a transparent point where there is no color information, that is, a region where there is no object.

[0079] Figure 5 The shown depth image G50 includes a region C51 corresponding to the AR object C41 of the boy located at the upper left, a region C52 corresponding to the AR object C42 of the baby located at the lower right, and a region C53 corresponding to the OSD object C43 of the clock located at the lower left. In the following description, in the case where a distinction is made between an object to be displayed to a subject in a real space (in this example, the objects C41 and C42) and an object to be displayed on a screen virtually set in a virtual setting (in this example, the object C43), the object to be displayed to a subject in a real space is referred to as a virtual object (including a performance effect, etc.) (hereinafter, referred to as an AR object), and the object to be displayed on a screen virtually set on a predetermined plane in a real space is referred to as an OSD object. Note that, in the following description, the object to be displayed to a subject in a real space is referred to as an AR object, and the object to be displayed on a screen virtually set in a virtual setting is referred to as an OSD object. Figure 5 In the shown depth image G50, the blacked-out region R51 indicates a region where no depth information is added or the depth information of the farthest point is added.

[0080] In the depth image, the depth information added to each point expressed as a pixel can be depth information corresponding to an actual distance from the position of the user 107 in a real space to the position in a real space corresponding to each pixel.

[0081] In this specification, depth information corresponding to a distance of 500 mm from the user 107 is added to the region C53 corresponding to the OSD object C43 of the clock located at the lower left. Depth information corresponding to a distance of 1000 mm from the user 107 is added to the region C52 corresponding to the AR object C42 of the baby located at the lower right, and depth information corresponding to a distance of 2000 mm from the user 107 is added to the region C51 corresponding to the AR object C41 of the boy located at the upper left.

[0082] Note that the input image data 10 can include mapping information on the type of an image to be reproduced as a hologram.

[0083] In the present embodiment, the input image data 10 can include label information on the objects included in the object image. In Figure 4 and Figure 5 In the example shown in Figure 6 , the label information can be added to the regions C61 to C63 respectively corresponding to the AR object C41 of the boy, the AR object C42 of the baby, and the OSD object C43 of the clock. Hereinafter, the image on which the regions C61 to C63 to which the label information is added are mapped is referred to as a label image G60. Note that in Figure 6 , the white region R61 indicates a region to which the label information is not added.

[0084] In Figure 6 , the region C61 is a region to which the label information indicating the AR object C41 of the boy is added, the region C62 is a region to which the label information indicating the AR object C42 of the baby is added, and the region C63 is a region to which the label information indicating the OSD object C43 of the clock is added. Note that in Figure 6 , the regions C61 and C62 corresponding to the objects C41 and C42 to be superimposed on the subjects in the real space are indicated with diagonal hatching, and the region C63 corresponding to the on-screen object C43 is indicated with black.

[0085] In Figure 3 , the points (corresponding to pixels) constituting the input image data 10 input in step S101 (hereinafter, described as pixels) are grouped into one or more groups in step S102. The algorithm for grouping the pixels can be variously changed depending on the actual use case. In the present embodiment, since it is difficult to realize both high resolution and shallow depth of field, the pixels requiring high resolution and the pixels requiring shallow depth of field are grouped into different groups, and different processing is performed.

[0086] For example, for an on-screen object such as the OSD object C43 of the clock, it is considered that the user desires to be able to clearly view the on-screen object regardless of the distance at which the user's viewpoint is aligned. Therefore, it is desirable that the on-screen object has high resolution and deep depth of field.

[0087] On the other hand, it is desirable that the AR objects such as the AR object C41 and the AR object C42 are displayed at positions close to the subjects on the real space corresponding to the AR objects, and for this purpose, it is necessary to have the same depth of field as that of the subjects on the real space. That is, shallow depth of field is important for the AR objects.

[0088] As described above, it is required that the OSD object has a high resolution and a deep depth of field, and that the AR object has a shallow depth of field even at the expense of resolution. In this regard, in the present embodiment, the pixels of the input image data 10 are grouped into one or more groups based on the tag information. For example, the input image data 10 is classified into a group of OSD objects (hereinafter, referred to as an OSD object group) and a group of AR objects (hereinafter, referred to as an AR object group). Note that a transparent region in which no object exists does not need to be grouped, and thus can be omitted.

[0089] In Figure 3 Step S103, the phase adjustment unit 22 of the preprocessing unit 20 adds a phase value to all pixels of the input image data 10. At this time, in the pixels belonging to the AR object group and the pixels belonging to the OSD object group, phase values of different characteristics are added thereto so that the reproduced pixels have different characteristics. Note that the phase value can not be added to the pixels not belonging to the AR object group or the OSD object group.

[0090] As a method of adding a phase value to each pixel, for example, a repeated random phase (RRP) method can be used. In the RRP method, first, a patch (hereinafter, referred to as a random phase patch) of a random phase pattern to be assigned to a region of a unit called a predetermined size (for example, m pixels x n pixels) is generated. The random phase pattern can be, for example, a pattern in which the difference between the phase values added to adjacent pixels is a random value that is not a fixed value.

[0091] In the present example, a random phase patch having the largest phase difference π among the patches is assigned to a plurality of pixels included in one unit. This assignment of the random phase patch in units is performed for the entire range of the input image data 10. Figure 7 is a graph showing an example of a random phase patch assigned to one unit in a case where the unit size is 1 pixel x 32 pixels. In Figure 7 , the horizontal axis represents the number of pixels (the number of samples) arranged in the horizontal direction in the input image data, and the vertical axis represents the phase value added to each pixel.

[0092] The random phase patch of 32 pixels shown in Figure 7 is repeated for a horizontal column of pixels of the input image data 10. This repetition is applied to all columns of the input image data 10.

[0093] By using the square root of the color intensity of each pixel in the object image G40 shown in Figure 4 , the input image data 10 is converted from a real value image to a complex value image having an amplitude according to the newly added phase information.

[0094] Random phase patterns can scatter light, thus widening the region formed by the light on the SLM plane after wave propagation. That is, high-frequency phase information in the input image space scatters the light to widen the region formed by the light on the SLM plane, while low-frequency phase information scatters the light to narrow the region formed by the light on the SLM plane.

[0095] By further scattering the light pattern on the SLM plane, high-frequency phase helps to make the depth of field shallower. Therefore, it can be said that high-frequency phase patches are more suitable for AR object groups.

[0096] On the other hand, low-frequency phase helps increase depth of field by reducing the light pattern on the SLM plane. Therefore, it can be said that low-frequency phase patches are more suitable for OSD object groups.

[0097] from Figure 7 As can be seen from the random phase patch shown, there are various methods to reduce or increase the frequency of the phase component. Examples of methods to reduce the frequency of the phase component include: multiplying the phase component by a coefficient less than 1.0 (hereinafter referred to as the phase coefficient) to reduce the amplitude of the random phase patch; and filtering the phase component with a low-pass filter. In this embodiment, as... Figure 8 As shown, the following situation is illustrated: by multiplying a random phase patch by a phase coefficient of 0.25, a new phase patch with a frequency lower than [a certain value] is generated. Figure 7 The random phase patch shown is a random phase patch with a frequency of random phase patch.

[0098] When using, as Figure 8 The low-frequency random phase patch shown adds phase values ​​to each pixel belonging to the OSD object group, allowing for a clearer and deeper depth of field reproduction of OSD object C43.

[0099] Similarly, when using, as Figure 7 The high-frequency random phase patch shown adds phase values ​​to each pixel belonging to the AR object group, allowing AR objects C41 and C42 to be reproduced with a shallower depth of field and lower resolution.

[0100] exist Figure 3 In step S104, based on the use Figure 5 The depth information of regions C51 to C53 corresponding to each described object will be propagated to the SLM plane by the complex field generated by the phase adjustment unit 22 of the preprocessing unit 20 using the above rules.

[0101] Figure 9 This is a diagram illustrating the positional relationship between the hologram to be reproduced, the hologram display system, and the viewer (user) in this embodiment. Figure 9Among the three holograms 106a to 106c to be reproduced, the hologram 106a located at the position E1 corresponds to the OSD object C43 of the clock, the hologram 106b located at the position E2 corresponds to the AR object C42 of the baby, and the hologram 106c located at the position E3 corresponds to the AR object C41 of the boy.

[0102] The CGH calculation unit 30 obtains each point that appears as a pixel on the SLM plane by using a wave propagation formula. As such a wave propagation formula, for example, a Fresnel diffraction formula as shown in the following formula (1) can be used. In formula (1), z denotes a distance from each image (hologram) plane to be reproduced to the spatial light modulator 105.

[0103]

[0104] where

[0105] E(x', y', 0) is an electric field at an aperture,

[0106]

[0107] k is a wave number 2π / λ

[0108] i is an imaginary unit.

[0109] For example, in Figure 9 Among the three holograms 106a to 106c to be reproduced, the hologram 106a located at the position E1 corresponds to the OSD object C43 of the clock, the hologram 106b located at the position E2 corresponds to the AR object C42 of the baby, and the hologram 106c located at the position E3 corresponds to the AR object C41 of the boy.

[0110] Note that the wave propagation formula used for generating a hologram on the SLM plane is not limited to the wave propagation formula shown in formula (1). For example, a Rayleigh-Sommerfeld diffraction type or a Fraunhofer diffraction type based on other assumptions, or the like can also be used.

[0111] Pixels of different depths propagate into the SLM plane separately from each other and are accumulated in the same field on the SLM plane. An example of the simplest method of integrating fields of different distances into one field includes a method of integrating fields. However, based on actual use cases, a more complex method of integrating fields of different depths can be applied to the present embodiment.

[0112] In Figure 3 In step S105, the post-processing described above is performed on the integrated complex field generated in step S104. Note that in the post-processing, different processing is performed depending on the type of the spatial light modulator 105. In the present embodiment using a reflection-type spatial light modulator 105, a process of directly displaying the complex field can be used.

[0113] In a complex-valued SLM, the amplitude component is quantized in the range of 0 to 255, and the phase component in the range of 0 to 2π is quantized into discrete values of 255 levels. In the post-processing according to the present embodiment, the complex field is mapped to a signal that can be displayed by the SLM and quantized.

[0114] In the mapping of the amplitude information, the following equation (2) can be used. According to equation (2), the field U generated by the CGH calculation unit 30 is converted into a signal that can be displayed by the SLM.

[0115]

[0116]

[0117]

[0118] where

[0119] U(x, y) is a complex field to be displayed,

[0120] A(x, y) is an amplitude component of the field,

[0121] φ(x, y) is a phase component of the field,

[0122] M(x, y) is a quantized amplitude signal to be displayed by the SLM,

[0123] A max is a maximum value in A(x, y),

[0124] θ(x, y) is a quantized phase value to be displayed by the SLM, and

[0125] Floor() is a function that takes the integer part of a value.

[0126] In step S106 of Figure 3 , the signal passing through the spatial light modulator 105 is displayed. In the present example, as shown in Figure 9 , the OSD object C43 of the clock is displayed as a hologram 106a at a position E1 located at a distance of 500 mm from the user 107 who is at a position F, the AR object C42 of the baby is displayed as a hologram 106b at a position E2 located at a distance of 1000 mm from the user 107, and the AR object C41 of the boy is displayed as a hologram 106c at a position E3 located at a distance of 2000 mm from the user 107.

[0127] Figure 10 and Figure 11 are diagrams showing examples of a scene visible to the user depending on where the user is looking, i.e., where the user's point of attention is located.Figure 10 is a diagram showing a scene visible to the user in the case where the user is looking at the baby in front, and Figure 11 is a diagram showing a scene visible to the user in the case where the user is looking at the boy behind. Note that in Figure 10 and Figure 11 , it is assumed that the user directly views the real space via the optical see-through type AR HMD with respect to regions R101 and R111 corresponding to backgrounds other than the objects C41 to C43 shown in Figure 4

[0128] As described above, in the present embodiment, a high-frequency phase value is added to the AR objects C41 and C42, thereby setting a shallow depth of field. Therefore, as shown in the image G100 in Figure 10 , in the case where the user 107 is looking at the baby in front, the AR object C102 of the baby is displayed in a sharp manner, while the AR object C101 of the boy located behind is displayed in a blurred manner. On the other hand, as shown in the image G110 in Figure 11 , in the case where the user 107 is looking at the boy behind, the AR object C111 of the boy is displayed in a sharp manner, while the AR object C112 of the baby located in front is displayed in a blurred manner.

[0129] As described above, according to the present embodiment, the user 107 can view the AR objects with a visual effect similar to that of a subject present in the real space (e.g., distance perception). Accordingly, the realism of the AR objects can be enhanced, and thus the user experience can be improved.

[0130] On the other hand, in the present embodiment, a low-frequency phase value is added to the OSD object C43, thereby setting a deep depth of field. Therefore, as shown in Figure 10 and Figure 11 , the OSD objects C103 and C113 of the clock are displayed in a sharp manner to the user 107 regardless of where the user 107 is looking.

[0131] 1.4 Effects and Advantages

[0132] As described above, according to the present embodiment, since appropriate parameters can be set with respect to different points on the same image, the user experience can be improved.

[0133] Specifically, since the resolution and the depth of field can be freely set with respect to each object included in the same image, an object with a high resolution and an object with a shallow depth of field can be presented to the user at the same time. Therefore, natural depth blur can be reproduced in the object with a shallow depth of field, and sharp reproduction that is easy to view can be performed in the object with a high resolution, so that the user experience provided to the user can be improved.​

[0134] Note that the above-described embodiment is a flexible framework, and it is self-evident that the AR developer can freely adjust the resolution and the depth of field of the objects according to the use case of the system the AR developer is developing. In addition, in the above-described embodiment, a case in which the depth of field of each object is adjusted by assigning phase patches having different maximum phase differences to each group to which a pixel is assigned is shown, but the present application is not limited to this. Various modifications can be made, for example, so that phase patches having different phase patterns or phase patches having different phase patterns are assigned to each group.

[0135] 2. Second Embodiment

[0136] Next, the information processing apparatus, the information processing method, and the program according to the second embodiment will be described in detail with reference to the drawings. Note that in the following description, the same configurations as those of the above-described embodiment are denoted by the same reference numerals, and redundant description thereof will be omitted.

[0137] Figure 12 is a block diagram showing a schematic configuration example of an information processing apparatus according to the present embodiment. As shown in Figure 12 , the information processing apparatus 2 according to the present embodiment has a configuration in which the preprocessing unit 20 further includes an image analysis unit 23, similarly to the configuration of the information processing apparatus 1 described with reference to Figure 2 in the first embodiment.

[0138] In the present embodiment, unlike the first embodiment, the input image data 10 does not include a label image (see Figure 6 ). Instead, in the present embodiment, the preprocessing unit 20 includes the image analysis unit 23.

[0139] The image analysis unit 23 analyzes the input image data 10 by using machine learning of a neural network such as a convolutional neural network (CNN). For example, the image analysis unit 23 classifies the pixels of the input image data 10 into a plurality of classes by classification, and labels the pixels classified into each class. The label added to the pixels in this way can be used instead of the label information in the first embodiment.

[0140] As machine learning using a neural network such as a convolutional neural network (CNN), for example, a pyramid scene parsing network (PSPNet) or the like can be used. The PSPNet can classify the pixels of the input image data 10 into a plurality of classes by the method described with reference to Figure 6 in the first embodiment.

[0141] Other configurations, operations, and effects can be similar to those of the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0142] 3. Third Embodiment

[0143] Next, an information processing apparatus, an information processing method, and a program according to the third embodiment will be described in detail with reference to the drawings. Note that, in the following description, configurations identical to those of the above-described embodiments are denoted by identical reference numerals, and redundant descriptions thereof will be omitted.

[0144] Figure 13 is a block diagram illustrating a schematic configuration example of an information processing apparatus according to the present embodiment. As Figure 13 indicated, in a configuration similar to that of the information processing apparatus 1 described with reference to Figure 2 the first embodiment, the information processing apparatus 3 according to the present embodiment further includes an imaging device (imaging unit) 60, and the pre-processing unit 20 further includes a gaze tracking unit 24. Note that, in the present embodiment, similar to the second embodiment, the input image data 10 does not include a label image (see Figure 6 ).

[0145] The imaging device 60 images, for example, the user 107 who is a viewer. Image data obtained by imaging the user 107 (hereinafter, referred to as a user image) is input to the gaze tracking unit 24 of the pre-processing unit 20. On the other hand, the gaze tracking unit 24 detects a gaze direction of the user by analyzing the input user image.

[0146] In the present embodiment, the grouping unit 21 groups the pixels of the input image data 10 on the basis of the gaze information input from the gaze tracking unit 24. For example, the grouping unit 21 according to the present embodiment groups the pixels of the input image data 10 into two groups on the basis of whether the pixels correspond to an object currently being looked at by the user 107. One group is a group of objects currently being looked at by the user 107 (hereinafter, referred to as an attention group), and the other group is a group of objects not included in the objects currently being looked at by the user 107 (hereinafter, referred to as an inattention group).

[0147] Figure 14 is a diagram illustrating a portion of the input image data corresponding to the direction in which the user is looking, and Figure 15 is a diagram illustrating an example of an image (hereinafter, referred to as a grouped image) corresponding to, for example, the label image illustrated in Figure 6 , and is a diagram for explaining an example of grouping objects on the basis of the gaze direction illustrated in Figure 14 .

[0148] As Figure 14As shown, when user 107's line of sight V141 is pointing towards the AR object C142 of the baby, as Figure 15 As shown, pixels belonging to region C152 corresponding to AR object C142 are grouped into a group of interest, and for AR objects other than AR object C142, in this example, pixels belonging to region C151 corresponding to AR object C141 and pixels belonging to region C153 corresponding to OSD object C143 are grouped into a group of no interest.

[0149] As described above, when the regions corresponding to each object are grouped based on the user 107's gaze direction, the phase values ​​to be added are assigned to the pixels belonging to each group. Since this example is a case where user 107 is looking at a baby in front of them, low-frequency random phase patches are assigned to the pixels corresponding to AR object C142 (i.e., pixels belonging to the attention group), thereby displaying AR object C142 in a clear manner at high resolution.

[0150] For the pixels corresponding to AR object C141 and OSD object C143 that are grouped into the non-interest group, high-frequency random phase patches are assigned to achieve shallow depth of field, thereby displaying the pixels as blurred to the same extent as the subject in the real world.

[0151] Note that the methods for assigning phase values ​​to pixels belonging to the interest group and to pixels belonging to the non-interest group can be similar to those described above. Figure 8 The method described is omitted here, and therefore its detailed description will be omitted.

[0152] Figure 16 and Figure 17 This is a diagram illustrating an example of a scene that is visible to the user depending on the user's line of sight. Figure 16 It is a diagram showing the scene visible to the user when their line of sight is directed towards the baby in front of them, and Figure 17 This diagram illustrates the scene visible to the user when their line of sight is directed towards the boy behind them. Note that... Figure 16 and Figure 17 In the middle, assuming for all but one Figure 14 The background areas R161 and R171, which are outside of objects C141 to C143, allow users to directly view the real space through an optically perceptible AR HMD.

[0153] In such Figure 16 In the image G160 shown, where user 107's gaze is directed towards the baby in front, low-frequency phase information is added to the pixels grouped into the attention group and corresponding to the baby's AR object C142. Therefore, the baby's AR object C162, which user 107 is looking at, is displayed clearly with a deep depth of field.

[0154] On the other hand, for the pixels grouped in the non-focus group and corresponding to the AR object C141 of the boy and the OSD object C143 of the clock, the depth of field is set to be shallow by adding high-frequency phase information. Therefore, the AR object C161 of the boy and the OSD object C163 of the clock are displayed in a blurred manner.

[0155] On the other hand, as in Figure 17 the image G170 shown, in a case where the line-of-sight direction of the user 107 is directed toward the boy in the back, low-frequency phase information is added to the pixels grouped in the focus group and corresponding to the AR object C141 of the boy. Therefore, the AR object C171 of the boy that the user 107 is looking at is displayed in a clear manner with a deep depth of field.

[0156] On the other hand, for the pixels grouped in the non-focus group and corresponding to the AR object C142 of the baby and the OSD object C143 of the clock, the depth of field is set to be shallow by adding high-frequency phase information. Therefore, the AR object C171 of the baby and the OSD object C173 of the clock are displayed in a blurred manner.

[0157] Note that the OSD object C143 can always be displayed in a high resolution and a deep depth of field regardless of the line-of-sight direction of the user 107. In this case, the region C153 corresponding to the OSD object C143 can be grouped in a group (hereinafter, referred to as an OSD group) different from the focus group and the non-focus group.

[0158] Other configurations, operations, and effects can be similar to those of the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0159] 4. Fourth Embodiment

[0160] Next, an information processing apparatus, an information processing method, and a program according to the fourth embodiment will be described in detail with reference to the drawings. Note that, in the following description, configurations identical to those of the above-described embodiments are denoted by the same reference numerals, and redundant descriptions thereof will be omitted.

[0161] Figure 18 is a block diagram showing a schematic configuration example of an information processing apparatus according to the present embodiment. As Figure 18 shown, in a configuration similar to that of the information processing apparatus 1 described with reference to Figure 2 in the first embodiment, the information processing apparatus 4 according to the present embodiment further includes the imaging device 60, and the pre-processing unit 20 further includes the feature detection unit 25. Note that, in the present embodiment, similar to the second embodiment, the input image data 10 does not include a label image (see Figure 6 ).

[0162] For example, the camera device 60 images the surrounding environment of user 107, including the viewpoint of user 107. The feature detection unit 25 generates a feature map based on the image of the surroundings of user 107 captured by the camera device 60 (hereinafter referred to as the surrounding image), indicating which areas in real space are likely to be viewed by user 107. The areas likely to be viewed by user 107 are, for example, areas that might attract user 107's attention, and high feature values ​​can be set in these areas in the feature map generated by the feature detection unit 25.

[0163] Figure 19 This is a diagram illustrating an example of a surrounding image captured by a camera device, and Figure 20 This is a diagram illustrating an example of a feature map generated by the feature detection unit based on the surrounding image according to this embodiment.

[0164] like Figure 19 As shown, the viewing angle of the camera device 60 includes, for example, the viewing angle of user 107. The viewing angle of the camera device 60 is not necessarily wider than the viewing angle of user 107. Figure 20 As shown, the feature detection unit 25 generates a feature map based on the surrounding image input from the camera device 60.

[0165] In this example, for example, as in Figure 20 In the feature map G200 shown, for the relationship with Figure 19 The highest feature value is set for region C202 corresponding to region C192 of the infant in the surrounding image G190 shown, the next highest feature value is set for region C204 corresponding to region C194 of the female, and the next highest feature value is set for region C201 corresponding to region C191 of the male.

[0166] According to this embodiment, the grouping unit 21 groups AR objects based on the feature map G200 generated as described above, according to the positional relationship between regions C201, C202, and C204 and AR objects in the input image data 10. For example, as Figure 21 As shown, grouping unit 21 can group the pixels corresponding to the baby's AR object C212 in region C202 of feature map G200 into a group with the highest attention (hereinafter referred to as the high attention group), and can group the pixels corresponding to other objects (in this example, the boy's AR object C211 and the clock's OSD object C213) into a group with low attention (hereinafter referred to as the low attention group).

[0167] In order to consistently display the high-attention AR object C202 in a clear, high-resolution manner, the phase adjustment unit 22 references... Figure 8 The described method adds low-frequency phase values ​​to pixels that are grouped into high-interest groups.

[0168] On the other hand, in the normal state, in order to display the AR object C201 having low attention in a blurred manner without discomfort, the phase adjustment unit 22 adds a high-frequency phase value to the pixels grouped into the low-attention group by referring to a table shown in FIG. 12B. Figure 7 The described method adds a high-frequency phase value to the pixels grouped into the low-attention group. Note that the abnormal state contrary to the normal state can be a state in which the user 107 gazes at a subject in the real space corresponding to an AR object different from the AR object grouped into the high-attention group.

[0169] Figure 22 and Figure 23 is a diagram showing an example of a scene visible to the user depending on the direction of the line of sight of the user, Figure 22 is a diagram showing a scene visible to the user in a case in which the direction of the line of sight of the user is directed to the baby in front, and Figure 23 is a diagram showing a scene visible to the user in a case in which the direction of the line of sight of the user is directed to the boy behind. Note that in Figure 22 and Figure 23 , it is assumed that the user directly views the real space via the optical see-through type AR HMD with respect to the regions R221 and R231 corresponding to the background other than the objects C211 to C213 shown in Figure 21

[0170] In Figure 22 and Figure 23 , since a low-frequency random phase patch is assigned to the AR objects C222 and C232 of the baby having high attention, the AR objects C222 and C232 of the baby are displayed in a clear manner with a deep depth of field regardless of where the user 107 is looking.

[0171] On the other hand, since a high-frequency random phase patch is assigned to the AR objects C221 and C231 of the boy having low attention, the AR objects C222 and C232 of the boy are displayed with a shallow depth of field. Therefore, in a case in which the user 107 is looking at the boy, the AR object C231 of the boy is displayed in a clear manner as shown in Figure 23 , but in a case in which the user 107 is not looking at the boy but, for example, looking at the baby, the AR object C231 of the boy is displayed in a blurred manner as shown in Figure 22 .

[0172] For example, in a case in which the user 107 always highly attends to or needs to highly attend to a specific subject in the real space, the present embodiment is considered to be particularly effective. Even in this case, since the AR object of the subject to which the user 107 does not pay attention is displayed in a degree of clarity according to the position of the attention of the user 107 or the like, a user experience without discomfort can be provided. ​

[0173] Other configurations, operations, and effects can be similar to those of the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0174] 5. Fifth Embodiment

[0175] Next, an information processing apparatus, an information processing method, and a program according to a fifth embodiment will be described in detail with reference to the drawings. Note that in the following description, configurations identical to those of the above-described embodiments are denoted by identical reference numerals, and redundant descriptions thereof will be omitted.

[0176] For example, the information processing apparatus according to the present embodiment can have a configuration similar to that of the information processing apparatus 3 according to the above-described third embodiment. However, the information processing apparatus according to the present embodiment performs the following operations.

[0177] Figure 24 And Figure 25 is a diagram for illustrating an operation performed by the information processing apparatus according to the present embodiment, Figure 24 is a diagram showing a portion of input image data corresponding to a direction in which a user is looking, and Figure 25 is a diagram for illustrating an example of grouping objects based on a line-of-sight direction shown in Figure 24

[0178] As shown in Figure 24 In the present embodiment, the line-of-sight tracking unit 24 detects a line-of-sight direction of the user 107 based on an image acquired by the imaging device 60, similarly to the third embodiment.

[0179] On the other hand, as shown in Figure 25 The grouping unit 21 according to the present embodiment groups a depth image in the input image data 10 based on depth information for each pixel. In the present example, similarly to the above-described example, the object image G240 includes an AR object C241 of a boy, an AR object C242 of a baby, and an OSD object C243 of a clock, and different depth information is added to each object in the depth image G250. For example, depth information indicating a distance of 2000 mm from the user 107 is added to a region C251 corresponding to the AR object C241 of the boy, depth information indicating a distance of 1000 mm from the user 107 is added to a region C252 corresponding to the AR object C242 of the baby, and depth information indicating a distance of 500 mm from the user 107 is added to a region C253 corresponding to the OSD object C243 of the clock.

[0180] ​In this case, the grouping unit 21 groups the regions of the objects of the input image data 10 based on a region C251 corresponding to the AR object C241 of the boy, a region C252 corresponding to the AR object C242 of the baby, and a region C253 corresponding to the OSD object C243 of the clock in the depth image G250.

[0181] As described above, by using the depth information included in the input image data 10, the grouping unit 21 can easily group the pixels of the input image data 10.

[0182] Similarly to the above-described embodiment, the phase adjustment unit 22 adds a phase value corresponding to the line-of-sight direction V241 of the user 107 to the pixels belonging to each group based on each distance generated as described above.

[0183] Note that, in the present embodiment, the phase value of each grouped pixel is set using a lookup table shown in Table 1 below.

[0184] Table 1

[0185]

[0186] As shown in Table 1, in the lookup table, the maximum phase difference in the patch is defined for each distance from the subject located on the line-of-sight direction V241 of the user 107 (i.e., the subject the user is looking at). Note that the random phase patch assigned to a unit of a predetermined size can be the random phase patch described above with reference to Figure 8 , or a random phase patch obtained by adjusting the random phase patch based on the maximum phase difference specified from the lookup table (see Figure 7 , for example).

[0187] Therefore, as shown in Figure 24 , in a case where the user 107 is looking at the baby located at a distance of 1000 mm from the user, a random phase patch in which the maximum phase difference in the patch is set to 1π (see Figure 7 , for example) is assigned to the pixels belonging to the group corresponding to the AR object C241 of the boy located 1000 mm away from the baby, and a random phase patch in which the maximum phase difference in the patch is set to 1 / 4π (see Figure 8 , for example) is assigned to the pixels belonging to the group corresponding to the OSD object C243 of the clock located 500 mm away from the baby.

[0188] Note that the maximum phase difference in the patch 0π (i.e., a constant phase value with no phase difference) is assigned to the pixels belonging to the group corresponding to the AR object C242 of the baby the user 107 is looking at.

[0189] As described above, the phase difference corresponding to the distance from the subject being viewed by the user 107 is provided to each object based on the lookup table, so that the following user experience can be provided.

[0190] Figure 26 and Figure 27 This is a diagram used to illustrate the user experience provided to the user according to this embodiment. Figure 26 It is a diagram illustrating the scene visible to the user when the user is looking at the baby in front of them, and Figure 27 This is a diagram illustrating the scene visible to the user when the user is looking at the boy behind them. Note that in... Figure 26 and Figure 27 In the middle, assuming for all but one Figure 24 The background areas R261 and R271, which are outside of objects C241 to C243, allow users to directly view the real space through an optically perceptible AR HMD.

[0191] In such Figure 26 In the image G260 shown, where user 107's gaze is directed towards the baby in front, random phase patches of a relatively low frequency (no phase difference in this example) are assigned to the pixels corresponding to the baby's AR object C242. Therefore, the baby's AR object C262, which user 107 is looking at, is displayed in a clear manner with deep depth of field.

[0192] On the other hand, a high-frequency random phase patch with the maximum phase difference corresponding to the distance of each subject from the baby (or user 107) is assigned to the pixels corresponding to objects C241 and C243, which have different distances from user 107 than the distances from the baby to user 107 (in this example, the boy and the clock), thus setting the depth of field to shallow. For example, based on the lookup table in Table 1, a high-frequency random phase patch with a maximum phase difference of 1π corresponding to the range of distances +1000mm from the baby is assigned to the pixel corresponding to the AR object C241 of the boy, and based on the lookup table in Table 1, a high-frequency random phase patch with a maximum phase difference of 1 / 4π corresponding to the range of distances -500mm from the baby is assigned to the pixel corresponding to the OSD object C243 of the clock. Therefore, the objects of these subjects (AR object C261 of the boy and OSD object C263 of the clock) are displayed in a blurred manner.

[0193] On the other hand, in such Figure 27 In the image G270 shown, where user 107's gaze is directed towards the boy behind them, a relatively low-frequency (no phase difference in this example) random phase patch is assigned to the pixel corresponding to the boy's AR object C242. Therefore, the boy's AR object C271, which user 107 is looking at, is displayed in a clear manner with deep depth of field.

[0194] On the other hand, a high-frequency random phase patch with the maximum phase difference corresponding to the distance of each subject from the boy (or user 107) is assigned to the pixels corresponding to objects C242 and C243, which have different distances from user 107 than the distances from the boy to user 107 (in this example, the baby and the clock), thus setting the depth of field to shallow. For example, based on the lookup table in Table 1, a high-frequency random phase patch with a maximum phase difference of 1π corresponding to the range of distances from the boy to -1000mm is assigned to the pixels corresponding to the AR object C242 of the baby, and based on the lookup table in Table 1, a high-frequency random phase patch with a maximum phase difference of 2 / 3π corresponding to the range of distances from the baby to -1500mm is assigned to the pixels corresponding to the OSD object C243 of the clock. Therefore, the objects of these subjects (AR object C272 of the baby and OSD object C273 of the clock) are displayed in a blurred manner.

[0195] Other configurations, operations, and effects may be similar to those of the embodiments described above, and therefore detailed descriptions thereof will be omitted here.

[0196] 6. Hardware Configuration

[0197] For example, it can be achieved by having, as Figure 28 The computer 1000 with the configuration shown implements the information processing apparatus, its modifications and application examples according to the above embodiments. Figure 28 This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of the information processing apparatus according to the above embodiments. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, hard disk drive (HDD) 1400, communication interface 1500, and input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0198] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 develops programs stored in ROM 1300 or HDD 1400 in RAM 1200 and executes processing corresponding to various programs.

[0199] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) program executed by CPU 1100 when computer 1000 starts up, and programs that depend on the hardware of computer 1000.

[0200] The HDD 1400 is a computer-readable recording medium that non-transitorily records a program executed by the CPU 1100, data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records a projection control program according to the present disclosure as an example of program data 1450.

[0201] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from other devices or transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0202] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface that reads a program or the like recorded in a predetermined recording medium (medium). For example, the medium is an optical recording medium such as a digital versatile disc (DVD) or a phase-change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.

[0203] For example, in a case where the computer 1000 is used as an information processing device according to the above-described embodiments, the CPU 1100 of the computer 1000 realizes the functions of the pre-processing unit 20, the CGH calculation unit 30, and the post-processing unit 40 by executing the program loaded on the RAM 1200. In addition, the HDD 1400 stores a program or the like according to the present disclosure. Incidentally, the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs can be acquired from other devices via the external network 1550.

[0204] Although each of the embodiments of the present disclosure has been described above, the technical scope of the present disclosure is not limited to each of the above-described embodiments, and various modifications can be made without departing from the gist of the present disclosure. Furthermore, the components can be appropriately combined in different embodiments and modifications.

[0205] The effects in each of the embodiments described in this specification are merely examples and are not limiting, and there can be additional effects.

[0206] Incidentally, the present technology can also be configured as follows. (1)

[0208] An information processing apparatus comprising:

[0209] a grouping unit that groups a plurality of pixels that constitute one or more objects included in one piece of image data into one or more groups;

[0210] a phase adjusting unit that assigns a phase pattern having a different phase difference for each of the one or more groups to the plurality of pixels; and

[0211] a calculation unit that generates hologram data from image data to which the phase pattern is added. (2)

[0213] The information processing apparatus according to (1), wherein the phase adjusting unit assigns a phase pattern having a different maximum phase difference for each of the one or more groups to the plurality of pixels. (3)

[0215] The information processing apparatus according to (1) or (2), wherein

[0216] the image data includes label information regarding the one or more objects, and

[0217] the grouping unit groups the plurality of pixels into the one or more groups based on the label information. (4)

[0219] The information processing apparatus according to (1), further comprising:

[0220] an image analysis unit that analyzes the image data to add a label to each pixel that constitutes the image data, wherein

[0221] the grouping unit groups the plurality of pixels into the one or more groups based on the label added by the image analysis unit. (5)

[0223] The information processing apparatus according to (1), further comprising:

[0224] a line-of-sight tracking unit that detects a line-of-sight direction of a user, wherein

[0225] the grouping unit groups pixels that constitute an object of the one or more objects that is associated with a subject located in the line-of-sight direction in a real space into a first group, and groups pixels that constitute an object that is not associated with a subject located in the line-of-sight direction into a second group. (6)

[0227] The information processing apparatus according to (1), further comprising:

[0228] an imaging unit that images a surrounding environment of a user; and

[0229] a feature detection unit that detects a feature of a subject present around the user by analyzing image data acquired by the imaging unit, wherein

[0230] the grouping unit groups the plurality of pixels constituting the one or more objects associated with the subject into the one or more groups based on the feature of the subject detected by the feature detection unit. (7)

[0232] The information processing apparatus according to (1), wherein

[0233] the image data includes depth information of each pixel constituting the image data, and

[0234] the grouping unit groups the plurality of pixels into the one or more groups based on the depth information. (8)

[0236] The information processing apparatus according to (7), further comprising:

[0237] a line-of-sight tracking unit that detects a line-of-sight direction of a user;

[0238] a storage unit that stores a look-up table of a correspondence relation between a holding distance and a phase difference, wherein

[0239] the phase adjustment unit determines the phase difference to be added to the phase pattern of the pixels belonging to each of the one or more groups by referring to the look-up table based on a distance between a subject associated with an object constituted by the pixels grouped into each of the one or more groups and a subject located in the line-of-sight direction in a real space. (9)

[0241] The information processing apparatus according to any one of (1) to (7), wherein

[0242] the phase adjustment unit

[0243] assigns a first phase pattern having the phase difference as a first phase difference to pixels grouped into a first group by the grouping unit, and

[0244] assigns a second phase pattern having a second phase difference greater than the first phase difference to pixels grouped into a second group. (10)

[0246] The information processing apparatus according to any one of (1) to (9), in which

[0247] The phase pattern has a predetermined pixel size, and

[0248] The phase adjustment unit assigns the phase pattern in units of a unit obtained by dividing the image data into the predetermined pixel size. (11)

[0250] The information processing apparatus according to any one of (1) to (10), in which the phase pattern is a random phase pattern in which a difference between phase values added to adjacent pixels is a random value. (12)

[0252] The information processing apparatus according to any one of (1) to (11), in which the one or more objects include a first object associated with a subject in a real space and a second object displayed on a virtual screen provided in the real space. (13)

[0254] The information processing apparatus according to (12), in which the phase adjustment unit assigns a low-frequency phase pattern having a small phase difference to pixels corresponding to the second object. (14)

[0256] An information processing method including:

[0257] grouping a plurality of pixels constituting one or more objects included in one piece of image data into one or more groups;

[0258] assigning a phase pattern having a different phase difference for each of the one or more groups to the plurality of pixels; and

[0259] generating hologram data for displaying a hologram from image data to which the phase pattern is added. (15)

[0261] A program for causing a computer to generate hologram data for displaying a hologram from image data:

[0262] grouping a plurality of pixels constituting one or more objects included in one piece of image data into one or more groups;

[0263] assigning a phase pattern having a different phase difference for each of the one or more groups to the plurality of pixels; and

[0264] Hologram data is generated from image data to which the phase pattern is added. (16)

[0266] A hologram display system includes:

[0267] The information processing apparatus according to any one of (1) to (13); and

[0268] A display apparatus displays a hologram for a user based on the hologram data generated by the information processing apparatus. (17)

[0270] The hologram display system according to (16), wherein

[0271] The information processing apparatus further includes a post-processing unit that converts the hologram data into a hologram signal that can be stereoscopically displayed on the display apparatus; and

[0272] The display apparatus includes a spatial light modulator that modulates light output based on the hologram signal to display the hologram for the user. (18)

[0274] The hologram display system according to (17), wherein the spatial light modulator is a reflection-type spatial light modulator.

[0275] List of Reference Signs

[0276] 1, 2, 3, 4 Information processing apparatus

[0277] 10 Input image data

[0278] 20 Pre-processing unit

[0279] 21 Grouping unit

[0280] 22 Phase adjustment unit

[0281] 23 Image analysis unit

[0282] 24 Line-of-sight tracking unit

[0283] 25 Feature detection unit

[0284] 30 CGH calculation unit

[0285] 40 Post-processing unit

[0286] 50 Display apparatus

[0287] 60 Imaging device

[0288] 101 Light source

[0289] 102, 103 lenses

[0290] 104 beam splitter

[0291] 105 spatial light modulator (SLM)

[0292] 106, 106a, 106b, 106c hologram

[0293] 107 user

Claims

1. An information processing apparatus, comprising: A grouping unit that groups multiple pixels constituting one or more objects in a single image data set into two or more groups; A phase adjustment unit, which assigns a phase pattern to the plurality of pixels having a different phase difference for each of the two or more groups; as well as A computing unit that generates holographic data based on image data with the added phase pattern.

2. The information processing apparatus according to claim 1, wherein, The phase adjustment unit assigns a phase pattern to the plurality of pixels, which has a different maximum phase difference for each of the two or more groups.

3. The information processing apparatus according to claim 1, wherein... The image data includes tag information about the one or more objects, and The grouping unit divides the plurality of pixels into two or more groups based on the label information.

4. The information processing apparatus according to claim 1, further comprising: An image analysis unit analyzes the image data to add labels to each pixel constituting the image data, wherein The grouping unit divides the plurality of pixels into two or more groups based on the labels added by the image analysis unit.

5. The information processing apparatus according to claim 1, further comprising: An eye-tracking unit detects the user's gaze direction, wherein... The grouping unit groups the pixels of objects that are associated with a subject in the real space located in the line of sight direction into a first group, and the pixels of objects that are associated with a subject not located in the line of sight direction into a second group.

6. The information processing apparatus according to claim 1, further comprising: An imaging unit that images the user's surrounding environment; and The feature detection unit analyzes image data acquired by the imaging unit to detect features of subjects present in the user's vicinity. The grouping unit, based on the features of the subject detected by the feature detection unit, groups the plurality of pixels constituting the one or more objects associated with the subject into the two or more groups.

7. The information processing apparatus according to claim 1, wherein... The image data includes depth information for each pixel that constitutes the image data, and The grouping unit divides the plurality of pixels into two or more groups based on the depth information.

8. The information processing apparatus according to claim 7, further comprising: An eye-tracking unit that detects the user's eye direction; The storage unit stores a lookup table that maintains the correspondence between distance and phase difference, wherein... The phase adjustment unit determines the phase difference of the phase pattern to be added to the pixels belonging to each of the two or more groups by referring to the lookup table, based on the distance between the subject associated with the object composed of pixels that are grouped into each of the two or more groups and the subject in real space located in the direction of the line of sight.

9. The information processing apparatus according to claim 1, wherein The phase adjustment unit The first phase pattern, with a phase difference equal to the first phase difference, is assigned to the pixels grouped into a first group by the grouping unit. The second phase pattern, whose phase difference is greater than the first phase difference, is assigned to the pixels grouped into the second group.

10. The information processing apparatus according to claim 1, wherein The phase pattern has a predetermined pixel size, and The phase adjustment unit allocates the phase pattern in units obtained by dividing the image data into units of the predetermined pixel size.

11. The information processing apparatus according to claim 1, wherein, The phase pattern is a random phase pattern in which the difference between the phase values ​​added to adjacent pixels is a random value.

12. The information processing apparatus according to claim 1, wherein, The one or more objects include a first object associated with a subject in real space and a second object displayed on a virtual screen set up in the real space.

13. The information processing apparatus according to claim 12, wherein, The phase adjustment unit assigns a low-frequency phase pattern with a small phase difference to the pixel corresponding to the second object.

14. An information processing method, comprising: Grouping multiple pixels that constitute one or more objects in a single image data set into two or more groups; Assign phase patterns with different phase differences for each of the two or more groups to the plurality of pixels; as well as Holographic data is generated from image data with the phase pattern added.

15. A computer program product comprising a program for causing a computer to perform the following operations, the computer generating holographic data for displaying a hologram based on image data: Grouping multiple pixels that constitute one or more objects in a single image data set into two or more groups; Assign phase patterns to the plurality of pixels, each having a different phase difference for each of the two or more groups; and Holographic data is generated from image data with the phase pattern added.

16. A hologram display system, comprising: The information processing apparatus according to claim 1; as well as A display device that displays a hologram to a user based on the hologram data generated by the information processing device.

17. The holographic display system according to claim 16, wherein... The information processing device further includes a post-processing unit, which converts the hologram data into a hologram signal that can be displayed three-dimensionally on the display device; and The display device includes a spatial light modulator that modulates light output based on the hologram signal to display the hologram to the user.

18. The holographic display system according to claim 17, wherein, The spatial light modulator is a reflective spatial light modulator.

19. A computer-readable storage medium having a computer-executable program stored thereon, wherein when the computer-executable program is executed by a processor, the processor performs an information processing method, comprising: Grouping multiple pixels that constitute one or more objects in a single image data set into two or more groups; Assign phase patterns with different phase differences for each of the two or more groups to the plurality of pixels; as well as Holographic data is generated from image data with the phase pattern added.

Images