Image Display System

Abstract

To provide an image display system capable of identifying a position and an attitude of a user with respect to a display with high accuracy and displaying an image of a virtual space viewed from the position.SOLUTION: In an image display system, a user space imaging part acquires a user space image in front of a screen on which the user may exist. A storage part holds screen arrangement data indicating a position, an attitude, and a shape of the screen in a reference coordinate space, imaging part arrangement data indicating a position and an attitude of the user space imaging part in the reference coordinate space, and three-dimensional data representing a three-dimensional object on the reference coordinate space. A processing part identifies a user viewpoint position on the basis of the imaging part arrangement data and the user space image, generates a display image of the three-dimensional object serving as an appearance when the three-dimensional object on the virtual space is viewed through the screen from the user viewpoint position on the basis of the user viewpoint position, the screen arrangement data, and the three-dimensional data, and displays the display image of the three-dimensional object on the screen of a display.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for displaying on a display an image that appears when a three-dimensional object in a virtual three-dimensional space is viewed from a user's position. [Background technology]

[0002] Patent Document 1 discloses a technology for displaying on a flat panel display an image in which a virtual object has a shape as seen from the position of the human eye. The three-dimensional image display device according to Patent Document 1 includes a display for displaying an image, a position measuring device for measuring the position of the human eye looking at the display, and a processing means for calculating a three-dimensional image in which the image to be displayed on the display has a shape as seen from the direction of the position of the human eye using information indicating the position of the human eye given by the position measuring device, and displaying the image on the display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 08-167048 Summary of the Invention [Problem to be solved by the invention]

[0004] By displaying an image of a pseudo-projected three-dimensional object in a virtual space that does not actually exist, so that the object appears to exist at the position when viewed from the user's viewpoint, the user can have the illusion that he or she is looking at the three-dimensional object in the virtual space that exists behind the display through a pseudo window created by the display. Using this technology, it is possible to provide the user with various experiences as a virtual reality experience device or an augmented reality experience device using a display. As an example, this display can be freely installed on a windowless wall of a hotel, house, office, etc., and the interior scenery can be improved as a pseudo window.

[0005] In virtual reality and augmented reality experiences, it is important that the images displayed on the display are natural and close to the actual field of view. In interior applications in hotels, homes, or offices, if the image displayed on the simulated window is unnatural, the degree of illusion to the user will be small, and it may not be possible to improve the interior scenery. For example, if the position of the user's viewpoint cannot be correctly detected or an image that can be seen from the user's viewpoint cannot be correctly generated, it will be impossible to display a projected image that matches the user's viewpoint, and the image displayed on the display will be unnatural to the user.

[0006] However, in the above-mentioned Patent Document 1, it is not considered to specify the relative position and posture between the display and the user with high accuracy. Therefore, it is possible that the position and posture of the user relative to the display cannot be specified correctly, and an image visible from the user's position cannot be displayed correctly on the display.

[0007] An object of the present invention is to provide a technique for identifying the position and orientation of a user relative to a display with high accuracy, and displaying on the display an image of a virtual space visible from that position. [Means for solving the problem]

[0008] An image display system according to one embodiment of the present invention comprises a display having a screen capable of displaying images, a user space imaging unit that images a user space in front of the screen where a user viewing an image displayed on the display screen may be present and acquires a user space image, a memory unit that holds screen layout data indicating the position, orientation and shape of the screen in a predetermined reference coordinate space, imaging unit layout data indicating the position and orientation of the user space imaging unit in the reference coordinate space, and three-dimensional data representing a three-dimensional object in the reference coordinate space, and a processing unit that identifies a user viewpoint position on the reference coordinate space based on the imaging unit layout data and the user space image acquired by the user space imaging unit, and generates a display image of the three-dimensional object based on the user viewpoint position, the screen layout data, and the three-dimensional data, such that the three-dimensional object appears as if it were viewed through the screen from the user viewpoint position in a virtual space, and displays the display image on the display. Effect of the Invention

[0009] According to the present invention, the screen layout data of the display, the imaging unit layout data of the user space imaging unit, and the three-dimensional data of the virtual space are set in the same reference coordinate space and calculations are performed, so that the user's position relative to the display can be identified with high accuracy and an image of the virtual space that is highly likely to be visible from that position can be displayed on the display. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of an image display system according to a first embodiment. [Diagram 2] 1 is a schematic external view of an image display system according to a first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining how the image display system according to the first embodiment displays an image according to a user's viewpoint position. [Figure 4] 4 is a diagram for explaining the angle of view of a user space imaging unit in the image display system according to the first embodiment. FIG. [Diagram 5] FIG. 4 is a diagram showing data recorded in a storage unit in the first embodiment. [Figure 6] 4 is a flowchart of the overall process of the image display system according to the first embodiment. [Figure 7] 4 is a flowchart of a viewpoint position calculation process performed by the image display system according to the first embodiment. [Figure 8] 4 is a flowchart of a gesture operation process of the image display system according to the first embodiment. [Figure 9] FIG. 11 is a block diagram of an image display system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0012] Fig. 1 is a block diagram of an image display system according to embodiment 1. Fig. 2 is a schematic external view of the image display system according to embodiment 1.

[0013] 1, an image display system 10 includes a display 11, two user space imaging units 12-1 and 12-2, a storage unit 13, and a processing unit 14. The user space imaging unit 12-1 and the user space imaging unit 12-2 are the same. In the following description, both or either of the user space imaging units 12-1 and 12-2 may be simply referred to as the user space imaging unit 12.

[0014] The display 11 is an output display device having a screen 111 capable of displaying an image. The display method is not particularly limited, and may be, for example, a liquid crystal display, a plasma display, an organic EL display, or an FED (field emission display). Alternatively, the display 11 may be a display device that projects an image onto a screen. In this case, the screen becomes the screen 111.

[0015] The user space imaging section 12 captures an image of a user space in front of the screen 111 where a user 90 viewing an image displayed on the screen 111 of the display 11 may be present, and obtains a user space image.

[0016] The usage of this screen display system 10 is not particularly limited, but in addition to using the surface screen 111 of the display 11 as a pseudo window to improve the interior scenery, it is possible to use it as a device for allowing users to experience virtual reality or augmented reality in entertainment facilities such as game centers and theme parks, or in stationary entertainment devices at home, or as an eye-catching object displayed indoors or outdoors as a work of art or advertisement. The display 11 of this image display system 10 can be used as a display device in virtual reality or augmented reality technology, and can allow users to experience realistic virtual reality or augmented reality in the same way as an HMD (head mounted display), without the hassle of the user having to wear a device on their head.

[0017] The installation location of the image display system 10 is not particularly limited, but for example, when used indoors in a hotel, office, house, etc., hotel guests, office workers, house residents, etc. are users 90, and the room in which the display 11 is installed is the user space. When used in an entertainment facility, etc., the person experiencing is the user 90, and the user space is the space used and the range in which the person experiencing can move (in this case, this includes various positions such as sitting, lying on their back, or lying on their stomach).

[0018] The storage unit 13 is either a volatile memory or a non-volatile memory, or both. The storage unit 13 holds screen layout data indicating the position, orientation, and shape of the screen 111 in a predetermined reference coordinate space, imaging unit layout data indicating the positions and orientations of the user space imaging units 12-1 and 12-2 in the reference coordinate space, and three-dimensional data expressing a virtual three-dimensional object in the reference coordinate space. The reference coordinate space is a space represented by a coordinate system (reference coordinate system) having a predetermined origin O that serves as a reference for calculation in this embodiment.

[0019] The reference coordinate space and its origin O may be set in any manner. For example, the reference coordinate space may be fixed to the screen 111, may be fixed to the user space imaging unit 12, may be fixed to both the screen 111 and the user space imaging unit 12, or may be fixed to neither.

[0020] In the example of FIG. 2, the reference coordinate space is a space of an orthogonal coordinate system having three axes, x, y, and z. s ,y s ,z s ) and posture (Yaw s ,Pitch s ,Roll s ) and the position of screen 111 (x m ,y m ,z m ), posture (Yaw m ,Pitch m ,Roll m ) and shape (Height m ,Width m ) are set in this reference coordinate space. In this example, the screen 111 is rectangular, and its shape is expressed by its height and width.

[0021] In this embodiment, as an example, the center position of the head of the user 90 in the reference space (x h ,y h ,z h ) is set as the user viewpoint position. The user viewpoint position may be the viewpoint position of the user 90 or the position of a point that can be regarded as the viewpoint of the user 90. It is not limited to the example of this embodiment. As another example, the center of both eyes of the user 90 may be set as the user viewpoint position. Furthermore, the center of a face area recognized by face recognition processing or person recognition processing, or the center of both recognized eyes may be set as the user viewpoint position.

[0022] In this embodiment, as an example, the center of the lens of the user space imaging unit 12 is set to the position (x s ,y s ,z s) is set as the center of the screen 111. m ,y m ,z m )

[0023] The processing unit 14 is a device that executes a predetermined process using data recorded in the storage unit 13, and is, for example, a processor that executes a process according to a software program. The processing unit 14 identifies a user viewpoint position on the reference coordinate space based on the imaging unit arrangement data and images acquired by the two user space imaging units 12-1 and 12-2. Furthermore, the processing unit 14 generates a display image of a three-dimensional object that looks like a three-dimensional object in a virtual space viewed from the user viewpoint position through the screen 111 based on the user viewpoint position, the screen arrangement data, and the three-dimensional data, and causes the display image to be displayed on the display 11. As a result, the display image on the display 11 looks as if the three-dimensional object exists in the virtual space from the user 90.

[0024] When identifying the user viewpoint position, the processing unit 14 may, for example, identify a person (user 90) from each of the user space images acquired at the same time by the two user space imaging units 12-1 and 12-2, identify the position where the head of the user 90 is captured in the user space image, and calculate the distance from the user space imaging units 12-1 and 12-2 to the head of the user 90 based on the imaging unit arrangement data and the parallax of the head of the user 90 in the two user space images. The user viewpoint position can be identified accurately with a simple configuration using the principle of triangulation.

[0025] However, the configuration and method for identifying the user's viewpoint position are not limited to those using the stereo sensor shown here, and any method can be used. As another example, a method may be used in which the user space imaging unit 12 irradiates an infrared laser with a dot pattern having many dots, and the distance from the user space imaging unit 12 to the part hit by each infrared laser is calculated from the image of the part hit by each infrared laser in the captured image. The infrared laser and / or its dot pattern have a predetermined spread angle, and in the part hit by the infrared laser dots, the size of the dots and / or the interval between the dots become larger as it is farther from the user space imaging unit 12. From the size of the dots and / or the interval between the dots, the distance from the user space imaging unit 12 to the part hit by the infrared laser can be calculated.

[0026] As yet another example, a method may be used in which the user space imaging unit 12 irradiates modulated infrared light, projects the infrared light reflected by an object and returns to an array sensor for each pixel, and calculates the distance from the user space imaging unit 12 of the object that reflected the infrared laser from the phase difference between the infrared light irradiated for each pixel and the infrared light received. As yet another example, there is a technology for calculating the depth of an object in an image from an image captured by a monocular imaging device, and this technology may be adopted in the user space imaging unit 12. As yet another example, there is a method for attaching a physical marker having a predetermined shape and / or size that is easy for a sensor to detect, or a marker of a light-emitting body that emits a predetermined light such as an infrared LED, to the user himself or an object held or worn by the user, and identifying the user's viewpoint position from the position of the marker, and this technology may be adopted in the user space imaging unit 12.

[0027] The display 11 and the user space imaging unit 12 according to this embodiment may be configured as devices each having a separate housing as shown in FIG. 2, or may be configured integrally in a single housing.

[0028] When the display 11 and the user space imaging unit 12 are configured as separate devices, a processing unit may be provided in each device, and the functions of the processing unit 14 described above may be shared between the processing units of the display 11 and the user space imaging unit 12. In this case, the processing unit in the display 11 device and the processing unit in the user space imaging unit 12 device communicate with each other via wired or wireless communication.

[0029] FIG. 3 is a diagram for explaining a state in which the image display system according to the first embodiment displays an image according to a user's viewpoint position. The user space in front of the screen 111 of the display 11 is a real space. The virtual space defined behind the screen 111 is displayed on the screen 111 as an image seen from a position 91 where the head of the user 90 is located through a pseudo window (hereinafter also referred to as a "pseudo window") created by the screen 111. A three-dimensional object in the virtual space is defined by three-dimensional data. In the example of FIG. 3, six trees are arranged side by side as three-dimensional objects. Note that here, the virtual space is defined behind the screen 111 in FIG. 3 for the sake of explanation, but the virtual space may be defined in front of the screen 111, or a space including the front and rear of the screen 111 may be defined as the virtual space. As a result, not only an image in which a three-dimensional object in the virtual space appears to be behind a virtual window created by the screen 111, but also an image in which a three-dimensional object in the virtual space appears to be protruding forward from the screen 111 can be displayed on the screen 111.

[0030] When the user 90 is at a position 91A in front of the screen 111, the field of view 92A of the virtual space seen through the pseudo window created by the screen 111 is wide, and all six trees are displayed on the screen 111 as if they are included in the field of view 92A (display 93A). When the user 90 moves from the position 91A in the z direction to a position 91B away from the screen 111, the field of view 92B of the virtual space seen through the pseudo window created by the screen 111 narrows, and only the entire three trees and parts of the trees on either side of them are displayed in the field of view 92B (display 93B). When the user 90 moves from the position 91A in the -x (negative x) direction to a position 91C, the field of view 92C of the virtual space seen through the pseudo window created by the screen 111 changes in the x direction. Only the three trees on the right side are included in the field of view 92C. In addition, in the field of view 92C, the screen 111 is viewed from an oblique direction rather than from the front, but the horizontal thickness of the tree seen through the pseudo window needs to be the same as when viewed from the front (display 93C'). For this reason, when displaying the tree on the screen 111, an image that has been appropriately stretched so that it looks like 93C' to the user is displayed (display 93C). In this way, as a process that makes the image look plausible and natural so that the user 90 has the illusion that a three-dimensional object in the virtual space exists there, in this embodiment, when generating an image to be displayed on the screen 111 of the display 11, a process (projective transformation) is performed to project the three-dimensional object in the virtual space defined in the three-dimensional data onto the screen 111 of the display 11, that is, onto a two-dimensional surface. As another method, each point of the three-dimensional data in the reference coordinate space may be projected onto the point where a straight line connecting each point and the user's viewpoint position intersects with the screen 111. As another processing method for generating an image to be displayed on the screen 111 of the display 11, arithmetic operations of specific matrices or values ​​based on empirical rules may be performed on the image or the three-dimensional parameters of the image.

[0031] The angle of view of the user space imaging section 12 is preferably 110° or more. In this embodiment, as an example, the angle of view of the user space imaging section 12 is assumed to be 110°. Below, the significance of the angle of view of the user space imaging section 12 being preferably 110° or more will be described.

[0032] Fig. 4 is a diagram for explaining the angle of view of the user space imaging unit in the image display system according to Example 1. Fig. 4 is a diagram showing a display 11 of the image display system 10 and a user 90 in a room as viewed from above. For ease of explanation, one user space imaging unit 12 is shown in Fig. 4.

[0033] Here, a situation is assumed in which a user 90 is working or studying in a private room with a pseudo window made of a display 11 with a width of 60 cm. The user 90 is near a wall and faces in a direction parallel to the wall. The width of the display 11 of 60 cm is a size that is familiar with the depth of a desk for working or studying and the width of general storage furniture used at home, and is one suitable size for the width of a pseudo window. In addition, when assuming the use as a stationary virtual experience device, a situation in which displays are arranged to surround the user is assumed, and in this case, the relative positional relationship between the user and the display arranged immediately next to the user is assumed to be as shown in FIG. 4. For example, in a situation in which three displays are arranged in a U-shape in front of the user, on the right side, and on the left side, the relative relationship between the user and the right and left displays is assumed to be as shown in FIG. 4.

[0034] When the head of the user 90 is in front of the display 11 (within the range of the width of the display 11 in front of the display 11), the user 90 will be clearly aware of the pseudo window created by the display 11 even if he or she is facing parallel to the screen 111, so it is preferable that the image on the display 11 be a natural image of the virtual space as seen from the viewpoint of the user 90.

[0035] 4, the body of the user 90 is positioned at a position that coincides with the edge of the display 11, and the user 90 faces a direction parallel to the display 11. Even in this state, the image display system 10 is required to be able to detect the user's viewpoint position of the user 90.

[0036] Assuming that the shoulder width of a human is 40 cm and the user space imaging unit 12 is positioned in the center of the width of the display 11, in order to be able to detect the user viewpoint position even if the user moves backwards from this position, for example by shifting the seat back, the user space imaging unit 12 needs to have an angle of view of 110° or more, as shown in FIG. 4.

[0037] As described above, according to this embodiment, the user space in front of the screen 111 is captured with an angle of view of 110° or more, and the user's viewpoint position is calculated from the obtained user space image, so that the user's viewpoint position can be recognized over a wide range in the user space and a pseudo window-like image visible to users in a wide measurement range can be displayed.

[0038] Furthermore, according to this embodiment, when a display 11 is used in which the width of the screen 111 is up to 60 cm, which is easily adapted to the depth of a desk or the width of storage furniture, if the display 11 has a viewing angle of 110° or more, the user's viewpoint position can be detected within a range in which the head of the user 90 can be present, taking into account a typical human shoulder width of 40 cm. This allows the display 11 to satisfactorily simulate a window up to about 60 cm wide.

[0039] Furthermore, according to this embodiment, the screen 111 and the user space imaging unit 12 are arranged in the same reference coordinate space, and the user viewpoint position and the display image are calculated in the reference coordinate space, so that the common design can be easily applied to systems in which the relative positions and / or orientations of the user space imaging unit 12 and the display 11 are different. For example, not only in a type of system in which the user space imaging unit 12 and the display 11 are separate devices and the positions at which they are arranged can be freely selected, but also in a type of system in which the user space imaging unit 12 and the display 11 are integrally configured and the relative positions and orientations are fixed, if there are multiple types with different screen sizes and shapes, the relative positions and orientations of the user space imaging unit 12 and the display 11 differ for each type. The common design can be applied to these different types.

[0040] Furthermore, when multiple displays 11 are used in the same room, no matter what positions and angles the displays 11 are installed at, the screens 111 of the multiple displays 11 can be arranged in the same reference coordinate space, and the images to be displayed on the screens 111 can be calculated in that reference coordinate space using a common calculation method. Also, the images of the multiple displays 11 in the same room can be made continuous with high accuracy. This provides a high visual effect in improving the interior scenery of hotels, homes, offices, etc.

[0041] In this embodiment, the angle of view of the user space imaging unit 12 is 110° or more, the width of the screen 111 is 60 cm or less, and the imaging direction of the user space imaging unit 12 is perpendicular to the screen 111. However, if the angle of view of the user space imaging unit 12 is 110° or more, it is possible to detect the user's viewpoint position well even in a system with other conditions. For example, if the angle of view is 110° or more, it is sufficiently practical for a screen width of up to about 100 cm. In addition, it is also possible to obtain better results by setting the imaging direction of the user space imaging unit 12 to a direction appropriately shifted up, down, left, and right from the direction perpendicular to the screen 111, taking into consideration the relative position of the user space imaging unit 12 with respect to the display 11, the expected installation locations of the display 11 and the user space imaging unit 12, etc. In other words, it is possible to detect the user's viewpoint position in a wider range of the user space.

[0042] In this embodiment, the processing unit 14 detects a predetermined gesture made by the user 90 from the time-series user space images and executes a predetermined process corresponding to the gesture. The user space imaging unit 12 acquires user space images with a wide angle of view, and the processing unit 14 detects a predetermined gesture from the user space images and executes a predetermined process corresponding to the gesture, so that a desired operation can be performed by a gesture operation by the user 90 within a practical range even without a remote controller for operation.

[0043] The operation of the image display system 10 will now be described in detail.

[0044] Fig. 5 is a diagram showing data recorded in the storage unit in the first embodiment. Referring to Fig. 5, the storage unit 13 records screen position data 21, screen attitude data 22, screen shape data 23, imaging position data 24, imaging attitude data 25, three-dimensional data 26, user viewpoint position data 27, and display image data 28. The above-mentioned screen layout data includes the screen position data 21, the screen attitude data 22, and the screen shape data 23 referred to here. The above-mentioned imaging unit layout data includes the imaging position data 24 and the imaging attitude data 25 referred to here.

[0045] The screen position data 21 is the position (x m ,y m ,z m ) is the data representing the position (x m ,y m ,z m ) may be determined and set when the display 11 is installed, or the display 11 may be moved freely after installation, and the position information recorded in the memory unit 13 may be updated according to the movement. When multiple displays 11 are used, the position (x m ,y m ,z m ) is set.

[0046] The screen attitude data 22 is the attitude (Yaw m ,Pitch m ,Roll m ) is the data representing the orientation of the screen 111 (Yaw m ,Pitch m ,Roll m The Yaw may be determined and set when the display 11 is installed, or may be set as a fixed value in advance together with the installation conditions of the display 11 at the time of design or shipment. For example, the installation conditions are set such that the display 11 is fixed horizontally to the wall, and the posture (Yaw) when the display 11 is installed according to the installation conditions is m ,Pitch m ,Roll mThe display 11 may be rotated freely after installation, and the numerical value of the posture information recorded in the storage unit 13 may be updated constantly according to the rotation. When multiple displays 11 are used, the postures (Yaw m ,Pitch m ,Roll m ) is set.

[0047] The screen shape data 23 is the shape of the screen 111 (height m ,Width m ) is data representing the shape of the screen 111 (Height m ,Width m The height (Height) of the display 11 may be set as a fixed value in advance at the time of design or shipment, or the display 11 may be freely deformed after installation, and the numerical value of the shape information recorded in the storage unit 13 may be constantly updated according to the deformation. When multiple displays 11 are used, the shape (Height m ,Width m ) is set. The screen 111 is assumed to have a rectangular shape, and m is the height direction, Width is m is a numerical value of the length in the width direction, but the shape of the screen 111 is not limited to a rectangle. For example, the screen 111 may be an isosceles triangle, and its shape may be expressed by the length of the lower base and the height. Also, the screen 111 may be a trapezoid, and its shape may be expressed by the upper and lower bases and the height. Also, a curved screen may be used as the screen 111. If the screen has a rectangular plane curved with a specific curvature, its shape can be expressed by the height and width of the rectangle and the curvature or radius. Similarly, screens of various shapes such as a sphere, a prolate spheroid, an oblate spheroid, and a solid of revolution may be expressed by parameters that characterize each shape. For any screen, including those exemplified here, the shape can be defined in the reference coordinate space, and the image to be displayed can be calculated.

[0048] The imaging position data 24 is the position (x s ,y s ,zs ) is data representing the position (x s ,y s ,z s ) is determined and set when the user space imaging unit 12 is installed.

[0049] The imaging attitude data 25 is the attitude (Yaw s ,Pitch s ,Roll s The data represents the orientation (Yaw s ,Pitch s ,Roll s ) may be determined and set when the user space imaging unit 12 is installed, or may be determined together with the installation conditions of the user space imaging unit 12 and set as a fixed value in advance.

[0050] The three-dimensional data 26 is three-dimensional data that is the basis for generating an image to be displayed on the display 11, and is three-dimensional data that expresses a three-dimensional object in a virtual space as exemplified in FIG. 3. The three-dimensional data referred to here may be data that expresses a three-dimensional object in a three-dimensional space, and the specific method is not particularly limited. It may be data having stereoscopic information created by computer graphics, or data in which a still image or a moving image taken on a flat three-dimensional object with no thickness is attached. It may be data in which a still image or a moving image taken by a camera using a lens with a wide viewing angle of 180° or close to that, or a 360° camera, is attached to the surface of a spherical object with no thickness.

[0051] The user viewpoint position data 27 is data that represents the above-mentioned user viewpoint position. Since the user viewpoint position is repeatedly updated by the processing unit 14, the user viewpoint position data 27 is also updated each time. For example, if the frame rate of the display 11 is 30 fps and the user viewpoint position is updated at the same cycle, the user viewpoint position data 27 will be updated 30 times per second. The user viewpoint position data 27 stores data on the user viewpoint position in time series.

[0052] The display image data 28 is data of an image obtained by projecting three-dimensional data so that a three-dimensional object appears to be in the space beyond the pseudo window of the screen 111 from the user's viewpoint position, as described above, and is data of the image displayed on the screen 111. When there are multiple displays 11, data to be displayed on the screen 111 of each display 11 is recorded as display image data 27. The data of the image displayed on the screen 111 is repeatedly updated by the processing unit 14, and therefore the display image data 28 is also updated each time. For example, if the frame rate of the display 11 is 30 fps, the display image data 28 is updated 30 times per second.

[0053] Fig. 6 is a flowchart of the overall processing of the image display system according to the embodiment 1. Each process in Fig. 6 is executed by the processing unit 14. By the processing unit 14 periodically repeating the processing in Fig. 6, the image displayed on the display 11 changes over time and is recognized as a moving image by the user 90. For example, if the frame rate of the image displayed on the display 11 is 30 fps, the processing unit 14 repeats the processing shown in Fig. 6 30 times per second.

[0054] 6, the processing unit 14 first executes a viewpoint position calculation process (step S101). The viewpoint position calculation process is a process of calculating the above-mentioned user viewpoint position and recording it as user viewpoint position data 26 in the storage unit 13. The details of the viewpoint position calculation process will be described later with reference to FIG.

[0055] Next, the processing unit 14 executes an image generation process (step S102). The image generation process is a process of generating an image of a three-dimensional object seen through the screen 111 from the user's viewpoint position indicated by the user's viewpoint position data 26 recorded in the storage unit 13, and recording the data in the storage unit 13. When there are multiple displays 11, the processing unit 14 generates an image to be displayed on the screen 111 of each display 11, and records the data in the display image data 27.

[0056] Next, the processing unit 14 executes an image display process (step S103). The image display process is a process for displaying an image of a three-dimensional object visible through the screen 111 on the display 11 based on the display image data 27 recorded in the storage unit 13. When there are multiple displays 11, the processing unit 14 causes each display 11 to display data of an image of a three-dimensional object visible through the screen 111 of each display 11.

[0057] For example, as shown in FIG. 2, when the display 11 and the stereo sensor having the user space imaging units 12-1 and 12-2 are configured as separate devices, step S101 may be performed by a processing unit of the stereo sensor, and steps S102 and S103 may be performed by a processing unit of the device that configures the display 11.

[0058] The processing unit 14 according to this embodiment may generate a display image of a three-dimensional object seen through the screen 111 from the user's viewpoint by applying projective transformation to the three-dimensional data 25. By projective transformation, the three-dimensional object of the three-dimensional data 25 is mathematically projected onto the screen taking into account the user's viewpoint, so that the three-dimensional object is displayed as a plausible and natural image even when the user 90 views the screen 111 from an oblique direction.

[0059] The processing unit 14 may also perform a process of removing noise contained in the user viewpoint position based on the user space images in time series up to the present obtained by the user space imaging unit 12, or the user viewpoint position, or both. If the user viewpoint position is calculated using the result of image processing of the most recent captured user space image as is and an image of a three-dimensional object visible from the user viewpoint position is generated, the user viewpoint position may be calculated as a position that is suddenly significantly shifted due to various noise elements that occurred only at that time point, and as a result, the displayed image may be blurred for each frame. According to this configuration, noise in the user viewpoint position is removed based on past user space images and user viewpoint positions, and then the current user viewpoint position is determined, so that blurring of the displayed image can be suppressed. As a method of removing noise, there is a method of performing noise determination based on data of past user space images and performing noise removal of the current space image based on the determination result. There is also a method of updating (or correcting) the current user viewpoint position using a Kalman filter based on data of the past user viewpoint position and data of the current user viewpoint position.

[0060] FIG. 7 is a flowchart of a viewpoint position calculation process performed by the image display system according to the first embodiment.

[0061] Referring to FIG. 7, the processing unit 14 first executes a feature search in an image (step S201). The feature search detects the head of a human (user 90) from an image of either the user space imaging unit 12-1 or 12-2. The process of detecting the human head from an image can be performed, for example, by using face recognition technology. In this case, a method of recognizing a face pattern based on the contrast of brightness using Haar-like features may be used. In addition, a method of recognizing the face position using a learning device (trained artificial intelligence program) that directly recognizes a user from an image trained by deep learning or machine learning, or a method of directly recognizing the center position of the head from the body skeleton may be used.

[0062] Next, the processing unit 14 executes an image matching process (step S202). The image matching process is a process of comparing the images captured by the user space imaging units 12-1 and 12-2 with each other, and identifying a location in one image that includes the same feature part as the user 90 searched for and recognized in the other image in step S201.

[0063] Next, the processing unit 14 specifies pixel coordinates of characteristic parts in both images (step 203). Here, the processing unit 14 acquires pixel coordinates in the image for each of the characteristic parts in the image captured by the user space imaging unit 12-1 and the image captured by the user space imaging unit 12-2.

[0064] Next, the processing unit 14 associates each pixel with three-dimensional coordinates based on the result of the image matching process (step S204). If the pixel coordinates of the location where the same object is captured in the two images are specified by the process of step S203, the three-dimensional coordinates of the location can be specified using the principle of triangulation.

[0065] If the identified location is the head of the user 90, the three-dimensional coordinates are identified as the user's viewpoint position. At this time, the processing unit 14, for example, determines the position of the center of the head as the head position (x h ,y h ,z h ) may be used. In this case, the radius of the head is estimated from the image, and the three-dimensional coordinates of the position that is the radius in the depth direction from the surface at the center of the head are taken as the head position. Specifically, the radius is added to the z value of the three-dimensional coordinates of the surface at the center of the head.

[0066] As another example, the processing unit 14 may calculate the position of the center surface of the head in the image as the head position (x h ,y h ,z h In this case, the three-dimensional coordinates corresponding to the pixel where the center of the head is displayed are used as the head position.

[0067] As another example of a method for identifying the user's viewpoint position, the processing unit 14 detects an area having a feature resembling a person in an image, obtains the distance from the user space imaging units 12-1 and 12-2 to the feature area, and then calculates the center position of the feature area in the horizontal direction as X V ,Y ,is a position about 10 cm below the apex of the feature area in the vertical direction. V As the pixel coordinate (X V、 Y V ) and the distance to the feature area, a simple method of estimating the user's viewpoint may be used. This estimates the viewpoint to be 10 cm below the top of the head.

[0068] The head of the user 90 is not necessarily captured in both of the user space images captured by the two user space imaging units 12-1 and 12-2. Due to differences in the positions of the user space imaging units 12-1 and 12-2, or the influence of obstacles, the head of the user 90 may be captured in only one of the user space images captured by the two user space imaging units 12-1 and 12-2. In such a case, the user viewpoint position may be estimated. This allows the user viewpoint position to be identified over a wide range and used to generate an image.

[0069] For example, the processing unit 14 may record at least one of the image data acquired by the user space imaging units 12-1, 12-2 and information on the user viewpoint position calculated from the image, and if a specific user part (here, the head) related to the user viewpoint position is captured in images captured by multiple user space imaging units 12, the processing unit 14 may calculate the user viewpoint position based on the parallax of the multiple images, and if a user part is captured in only an image captured by one user space imaging unit 12, the processing unit 14 may estimate the user viewpoint position based on that one image and data from a previous image or a previous user viewpoint position.

[0070] In this embodiment, when the user 90 moves violently or when the processing unit 14 cannot detect the feature due to the influence of the brightness of the room, etc., it may be possible that the processing unit 14 cannot temporarily calculate the user viewpoint position of the user 90 from the images captured by the user space imaging units 12-1 and 12-2. In that case, the processing unit 14 may estimate the current user viewpoint position based on the past time-series user viewpoint positions stored in the storage unit 13 as the user viewpoint position data 27. Specifically, data on the movement speed and movement acceleration of the user 90 and three-dimensional vectors indicating their directions may be recorded from the history of the past user viewpoint positions, and the current user viewpoint position may be estimated from the data. For example, if there is no obstacle ahead in the movement direction, the speed may be changed from the latest calculated movement speed to the latest calculated movement acceleration, and the position moved from the latest calculated user viewpoint position to the present may be set as the current user viewpoint position.

[0071] In addition, in this embodiment, the processing unit 14 may correct the user viewpoint position and generate an image of a three-dimensional object seen through the screen 111 from the corrected user viewpoint position. The accuracy of the user viewpoint position can be improved. For example, a plurality of stereo sensors having user space imaging units 12-1 and 12-2 may be provided, and the user viewpoint position may be estimated based on a least squares criterion from a plurality of user viewpoint positions acquired by the stereo sensors. In addition, the user viewpoint position may be estimated based on a least squares criterion from two or more user viewpoint positions obtained by any combination of two or more from a stereo sensor having two user space imaging units 12-1 and 12-2 and a sensor using infrared rays or a sensor that measures the user position using a marker, which are exemplified as other sensors.

[0072] In this embodiment, the user 90 can operate the image display system 10 by gestures. Fig. 8 is a flowchart of a gesture operation process of the image display system according to the first embodiment.

[0073] The processing unit 14 monitors whether a predetermined activation gesture is made based on the user space image updated in time series (step S301). The activation gesture is a preliminary gesture for giving an instruction to the image display system 10. The specific gesture is not particularly limited, but may be, for example, a gesture such as placing a hand on a head. While the activation gesture is not detected, even if an instruction gesture for instructing a predetermined process is made, the image display system 10 does not execute the process.

[0074] When the activation gesture is detected, the processing unit 14 then activates a timer (step S302). This timer measures the time for which the activation gesture makes the instruction gesture valid. When the activation gesture is detected, the processing unit 14 may present to the user 90 that the activation gesture has been detected by displaying a predetermined instruction acceptance indication on the screen 111 of the display 11. Furthermore, the processing unit 14 may continue to display the instruction acceptance indication on the display 11 until the activation gesture is detected or the timer times out.

[0075] Next, the processing unit 14 monitors whether a pointing gesture is made based on the user space image updated in time series (step S303). The pointing gesture is a gesture for instructing a predetermined process as described above. There may be multiple types of processes that can be instructed by a gesture. In that case, a different pointing gesture may be defined for each process.

[0076] While the instruction gesture is not detected (NO in step S303), the processing unit 14 monitors whether the timer has timed out (step S305), and monitors the instruction gesture until the timer has timed out.

[0077] If the timer times out before the instruction gesture is detected (YES in step S305), the processing unit 14 returns to step S301 and monitors the activation gesture.

[0078] If a pointing gesture is detected before the timer times out (YES in step S303), the processing unit 14 executes a process corresponding to the pointing gesture (step S304).

[0079] As described above, when the processing unit 14 detects a predetermined instruction gesture after detecting a predetermined activation gesture from the time-series user space images, the processing unit 14 executes a predetermined process corresponding to the instruction gesture. Since the process is executed when there is a two-stage gesture, it is possible to prevent a process unintended by the user 90 from being executed by mistake.

[0080] In this embodiment, an example in which a two-step gesture is adopted is shown, but the present invention is not limited to this. As another example, the image display system 10 may be operated by a one-step gesture. Even in this case, it is possible to suppress erroneous operation by using a gesture that is rarely performed in normal operations. Also, when a predetermined instruction gesture is detected after a two or more step prior gesture is detected, a process corresponding to the instruction gesture may be executed.

[0081] In addition, in this embodiment, an example in which the image display system 10 has one display 11 has been shown, but the present invention is not limited to this. As another example, the image display system 10 may have a plurality of displays 11. In this case, for example, the storage unit 13 holds screen arrangement data indicating the position, attitude, and shape of each of the screens 111 of the plurality of displays 11 in a reference coordinate space. In addition, the processing unit 14 generates a display image of a three-dimensional object for each of the plurality of displays 11 based on the screen arrangement data (screen position data 21 and screen attitude data 22) of the display 11 and the three-dimensional data 25 of the display 11, and displays each display image on the screen 111 of the display 11. Since the positions and attitudes of the screens 111 of all the displays 11 are arranged in the same reference coordinate space and calculations are performed, images to be displayed on all the displays 11 can be calculated by a common calculation method regardless of the positions and attitudes of the plurality of displays 11. Furthermore, since the positions and attitudes of the screens 111 of all the displays 11 are arranged in the same reference coordinate space and calculations are performed, the images of the plurality of displays 11 can be aligned and continuous with high accuracy. This provides a high visual effect in improving the interior scenery of hotels, homes, offices, etc.

[0082] Furthermore, the image display system 10 according to this embodiment may be provided with various additional functions.

[0083] For example, the image display system 10 may include a communication device that connects to a communication network such as the Internet. The image display system 10 may transmit information to a Web server or receive information from a Web server via the communication device. The operations of transmitting and receiving information may be performed by the above-mentioned gestures.

[0084] The image display system 10 may be connected to a personal computer and used as a monitor for displaying images from the personal computer (PC). The connection interface unit may be, for example, a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark) or a Video Graphics Array (VGA). Operations such as switching between the virtual window and the PC monitor mode may be performed by the above-mentioned gestures. EXAMPLES

[0085] In the first embodiment, an image display system 10 that displays an image of a virtual three-dimensional object is exemplified, but the present invention is not limited to this. As another example, an image in real space can be displayed on the display 11. In the second embodiment, an image display system that displays an image in real space is exemplified.

[0086] Fig. 10 is a block diagram of an image display system according to a second embodiment. Referring to Fig. 10, an image display system 30 has an image display device 31 and an image acquisition device 33. The image display device 31 and the image acquisition device 33 are connected via a communication line. For example, the image display device 31 may acquire three-dimensional data based on an actual image via a communication network, and the image acquisition device 33 may display the acquired data. Alternatively, the image display device 31 and the image acquisition device 33 may be connected via wired or wireless communication, and the image display device 31 may display an image based on an image acquired by the image acquisition device 33 in real time. The image display device 31 is the same as the image display system 10 shown in the first embodiment, and includes a display 11, two user space imaging units 12-1 and 12-2, a storage unit 13, and a processing unit 14. The image acquisition device 33 includes a display space imaging unit 34, a processing unit 35, and an audio acquisition unit 36.

[0087] The display space imaging unit 34 obtains an image of the display space, which is the real space to be displayed on the display 11.

[0088] The audio acquisition unit 36 ​​acquires audio in the display space.

[0089] The processing unit 35 generates three-dimensional data 25 by using the image data acquired by the display space imaging unit 34. Then, the processing unit 35 transmits the generated three-dimensional data and the audio data acquired by the audio acquisition unit 36 ​​to the image display device 31.

[0090] The method in which the processing unit 35 generates the three-dimensional data 25 from the real-life image acquired by the display space imaging unit 34 is not particularly limited. For example, the acquired planar real-life image may be left as it is, or may be divided into multiple images of a predetermined size, and the images may be arranged in a three-dimensional space as planar images. As the display space imaging unit 34, a camera using a lens with a wide viewing angle of 180° or close to it, or a 360° shooting camera, etc. may be used, and the image captured by the display space imaging unit 34 may be arranged in the three-dimensional space in a form of being attached to the inner surface of a spherical object in the three-dimensional space. In addition, a camera using light field technology in the display space imaging unit 34 acquires information on the incident direction and intensity of light, and the information is used to perform image processing, thereby obtaining data that captures the three-dimensional space itself, including depth information. This data may be arranged in a three-dimensional space. In addition, the real-life image may be three-dimensionalized and arranged in a three-dimensional space. A real-life image having depth information may be captured by multiple stereo cameras, and three-dimensional data may be created based on the real-life image and the depth information. In this case, the display space imaging section 34 is a collection of a plurality of individual imaging sections, and obtains images of the display space captured from a plurality of directions.

[0091] As described above, in this embodiment, the image capturing device 33 captures an image of the display space, which is the real space to be displayed, by the display space capturing unit 34. The processing unit 35 generates three-dimensional data using the data of the image captured by the display space capturing unit 34. Then, the image display device 31 uses this three-dimensional data to display on the display 11 an image of the three-dimensional space represented by the three-dimensional data as viewed from the position of the user 90 through the screen 111. Therefore, an image of the real space can be displayed on the display 11 as an image seen from the user's viewpoint position.

[0092] In this embodiment, the audio acquisition unit 36 ​​acquires audio in the display space. Then, the audio output unit 32 outputs the audio to the user space. Therefore, since not only the image but also the audio in the display space can be reproduced, the user 90 can visually and aurally sense the real space beyond the pseudo window created by the display 11.

[0093] The method of using the screen display system 30 according to this embodiment is not particularly limited, but possible uses include placing the image capture device 33 in a stadium or the like to simulate the view from the stands for watching an actual sports game using the image display device 31, and connecting the image display system 30 via a communication network to transmit and receive images and audio captured by the image capture device 33 to and from each other, thereby enabling real-time communication between remote locations using images and audio.

[0094] The above-mentioned embodiments are merely illustrative examples of the present invention, and are not intended to limit the scope of the present invention. Those skilled in the art can implement the present invention in various other forms without departing from the scope of the present invention. [Explanation of symbols]

[0095] 10...image display system, 11...display, 12...user space imaging unit, 13...storage unit, 14...processing unit, 21...screen position data, 22...screen attitude data, 23...screen shape data, 24...imaging position data, 25...imaging attitude data, 26...three-dimensional data, 27...user viewpoint position data, 28...display image data, 30...image display system, 31...image display device, 32...audio output unit, 33...image acquisition device, 34...display space imaging unit, 35...processing unit, 36...audio acquisition unit, 90...user, 111...screen

Claims

1. A display having a screen capable of displaying an image; a user space imaging unit that images a user space in front of the screen where a user who views an image displayed on the screen of the display may be present, and acquires a user space image; a storage unit that holds screen layout data indicating a position, orientation, and shape of the screen in a predetermined reference coordinate space, imaging unit layout data indicating a position and orientation of the user space imaging unit in the reference coordinate space, and three-dimensional data representing a three-dimensional object in the reference coordinate space; a processing unit that identifies a user viewpoint position on the reference coordinate space based on the imaging unit arrangement data and the user space image acquired by the user space imaging unit, generates a display image of the three-dimensional object based on the user viewpoint position, the screen arrangement data, and the three-dimensional data, such that the three-dimensional object appears as if viewed from the user viewpoint position in a virtual space via the screen, and displays the display image on the screen of the display; having the processing unit records at least one of data of an image acquired by the user space imaging unit and the user viewpoint position calculated from the image in the storage unit, and thereafter, when the image acquired by the user space imaging unit matches a predetermined condition, performs a process of estimating the user viewpoint position based on at least one of the time-series user viewpoint position recorded in the storage unit and the time-series image captured by the user space imaging unit. Image display system.

2. when the processing unit cannot identify a current user viewpoint position from the image acquired by the user space imaging unit, the processing unit estimates the user viewpoint position based on at least one of the time-series user viewpoint positions recorded in the storage unit and the time-series images captured by the user space imaging unit; 2. The image display system according to claim 1.

3. when the processing unit is in a state where it is temporarily unable to calculate a current user viewpoint position from an image acquired by the user space imaging unit, the processing unit estimates the user viewpoint position based on at least one of the time-series user viewpoint positions recorded in the storage unit and the time-series images captured by the user space imaging unit; 3. The image display system according to claim 2.

4. When the user moves vigorously in the image acquired by the user space imaging unit, the processing unit estimates the user's viewpoint position based on at least one of the time-series user's viewpoint position recorded in the storage unit and the time-series image acquired by the user space imaging unit.

3. The image display system according to claim 2.

5. when a feature amount of the user's head cannot be detected from the image acquired by the user space imaging unit, the processing unit estimates the user's viewpoint position based on at least one of the time-series user's viewpoint position recorded in the storage unit and the time-series images captured by the user space imaging unit.

3. The image display system according to claim 2.

6. the processing unit records data indicating a moving speed of the user's viewpoint position based on the time series of the user's viewpoint position recorded in the storage unit, and estimates the user's viewpoint position from the data.

2. The image display system according to claim 1.

7. the processing unit records data of a three-dimensional vector indicating a moving speed and a moving direction of the user's viewpoint position based on the time series of the user's viewpoint position recorded in the storage unit, and estimates the user's viewpoint position from the data.

7. The image display system according to claim 6.

8. The processing unit records data indicating a moving speed and a moving acceleration of the user's viewpoint position based on the time series of the user's viewpoint position recorded in the storage unit, and estimates the user's viewpoint position from the data.

7. The image display system according to claim 6.

9. the processing unit records data on a three-dimensional vector indicating a moving speed and a moving acceleration of the user's viewpoint position and their directions based on the time series of the user's viewpoint position recorded in the storage unit, and estimates the user's viewpoint position from the data.

9. The image display system according to claim 8.

10. the processing unit performs a process of removing noise contained in the user's viewpoint position based on at least one of the time-series user's viewpoint position recorded in the storage unit and the time-series image captured by the user space imaging unit.

2. The image display system according to claim 1.

Images