Display system and method for controlling same

Abstract

Provided is a display system that presents a three-dimensional image and an image of a virtual world to a user with unaided vision who changes the head position and posture.　The display system comprises: a display device; a robotic arm that supports the display device; a measurement unit that measures the viewpoint position of a user who observes a screen of the display device; and a control unit that, on the basis of the measurement result obtained by the measurement unit, controls driving of the robotic arm so that the screen of the display device can be observed from the user's viewpoint. The display system further comprises a moving device on which the robotic arm is mounted. The control unit controls the moving device as well.

Description

Display system and control method thereof

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to a display system that presents images to a user and a control method thereof.

[0002] Known display methods for presenting three-dimensional images or virtual world images include head-mounted displays, CAVE systems, and stereoscopic displays. For example, a head-mounted display that displays free-viewpoint images that track the movement of the viewer's head and body has been proposed (see Patent Document 1). CAVE is a system that projects binocular parallax images, in which images for the right and left eyes alternate at high speed, onto each wall of a room-sized cube, and allows a user to view these images using stereoscopic glasses such as liquid crystal shutter glasses (see Non-Patent Document 1). Stereoscopic displays include types in which the user observes binocular parallax images through stereoscopic glasses such as liquid crystal shutter glasses, and types in which the user can view stereoscopic images with the naked eye. Examples of the latter type of naked-eye stereoscopic displays include a lenticular lens system that combines a two-dimensional display device such as a liquid crystal display with a lenticular lens (see Patent Document 2, for example), and a parallax barrier system that combines a two-dimensional display device with a parallax barrier (see Patent Document 3, for example).

[0003] WO2015 / 098211 JP 2009-48116 A JP 2013-5135 A

[0004] Carolina Cruz-Neira, Daniel J. Sandin, Thomas A. DeFanti, Robert V. Kenyon and John C. Hart, “The CAVE: Audio Visual Experience Automatic Virtual Environment,” Communications of the ACM, No. 35(6), 1992, pp. 64-72)

[0005] An object of the present disclosure is to provide a display system and a control method thereof that presents three-dimensional images and images of a virtual world to a user with naked eyes.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and a first aspect thereof is a display system comprising: a display device; a robot arm supporting the display device; a measurement unit that measures the head of a user observing the screen of the display device; and a control unit that controls the drive of the robot arm based on the measurement results by the measurement unit so that the screen of the display device can be observed from the viewpoint of the user.

[0007] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is contained within a single housing. In other words, both a single device consisting of multiple parts or functional modules and a collection of multiple devices are considered "systems."

[0008] The display system according to the first aspect may further include a mobile device carrying the robot arm, and the control unit may control the movement of the mobile device so that, when the user moves, the screen of the display device is positioned so as to be observable from the user's viewpoint.

[0009] The control unit first calculates, based on the measurement results by the measurement unit, an optimal position and orientation where the normal direction of the screen of the display device coincides with the line of sight of the user and the screen of the display device fits perfectly within the field of view of the user.The control unit then calculates, by inverse kinematics calculation, the rotation angles of each joint axis of the robot arm for positioning the display device at the above position and orientation, and controls the drive of the robot arm.

[0010] The control unit may also predict future head movement based on past head movement of the user, calculate the position and posture of the display device that follows the predicted head movement of the user, and control the drive of the robot arm.

[0011] A second aspect of the present disclosure is a control method for a display system configured by supporting a display device with a robot arm, the control method comprising: a measurement step of measuring the head of a user observing the screen of the display device; and a control step of controlling the drive of the robot arm based on the measurement result in the measurement step so that the screen of the display device can be observed from the viewpoint of the user.

[0012] According to the present disclosure, it is possible to provide a display system and a control method thereof that presents three-dimensional images and images of a virtual world to a naked-eye user who changes the position and posture of his or her head.

[0013] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings.

[0015] FIG. 1 is a diagram showing a basic configuration of a display system according to the present disclosure. FIG. 2 is a diagram illustrating an example of how a robot arm 102 operates in accordance with a user's head position. FIG. 3 is a diagram illustrating an example of how a robot arm 102 operates in accordance with a user's head position. FIG. 4 is a diagram illustrating an example of how a robot arm 102 operates in accordance with a user's head position. FIG. 5 is a diagram illustrating a mechanism for displaying an image that follows the movement of a user's head on a display device 101. FIG. 6 is a flowchart showing the operation of a display system 100. FIG. 7 is a diagram showing another example configuration of a display system according to the present disclosure. FIG. 8 is a diagram illustrating an example of how a moving device 705 and a robot arm 702 operate in accordance with a user's movement and head position. FIG. 9 is a diagram illustrating an example of how a moving device 705 and a robot arm 702 operate in accordance with a user's movement and head position. FIG. 10 is a diagram illustrating an example of how a moving device 705 and a robot arm 702 operate in accordance with a user's movement and head position. FIG. 11 is a flowchart showing the operation of a display system 700. FIG. 12 is a diagram showing calculation of the position of the display device following the user's head movement. FIG. 13 is a diagram showing prediction of the position of the display device following the user's head movement. FIG. 14 is a diagram showing a user exploring a virtual space. FIG. 15 is a diagram showing an example of a screen display when the display system 700 is applied to virtual space exploration. FIG. 16 is a diagram showing an example of a screen display when the display system 700 is applied to virtual space exploration. FIG. 17 is a diagram showing an example of the configuration of a remote conference system in which the display system 700 is applied to each location. FIG. 18 is a diagram showing an example of the operation of the display system 700 applied to a remote conference. FIG. 19 is a diagram showing an example of the operation of the display system 700 applied to a remote conference. FIG. 20 is a diagram showing a user pushing a mobile device 705 forward using a handle 705A. FIG. 21 is a diagram showing a user pushing a handle 701A attached to the front of the display device 701 forward. FIG. 22 is a diagram showing an example of the configuration of an information processing device 2000.

[0016] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.

[0017] A. Overview B. Basic configuration and operation C. Other configuration examples and operation D. Calculation and prediction of head position D-1. Calculation of display device position following head movement D-2. Prediction of head position E. Application examples E-1. Searching virtual space E-2. Application to remote conferences E-3. Hand-pushing operation by user F. Configuration of information processing device

[0018] A. Overview Head-mounted displays, CAVE systems, and autostereoscopic displays are known as display methods for presenting three-dimensional images and virtual world images. Head-mounted displays have the drawback of being inconvenient for users to wear on their heads, and completely obstructing the view when worn, making it difficult to move between real and virtual spaces. CAVE systems also have the drawback of requiring large-scale equipment, resulting in high costs, and being disconnected from the outside world during use. Furthermore, typical autostereoscopic displays are only a few tens of inches in size, limiting the user's viewpoint to a small screen size.

[0019] Therefore, this specification proposes a display system having a movable display device according to the present disclosure. According to the display system according to the present disclosure, the position of the display device (screen) can be dynamically and adaptively changed in response to the movement of the user's viewpoint. Since the user does not need to wear anything on their head or body, use is hassle-free and they can freely move between real space and virtual space. Furthermore, the display system according to the present disclosure does not require large-scale facilities like a CAVE system.

[0020] B. Basic Configuration and Operation The basic configuration of a display system according to the present disclosure is shown in Fig. 1. The display system 100 shown in Fig. 1 includes a display device 101, a robot arm 102, a head position measurement unit 103, and a control unit 104.

[0021] The display device 101 is, for example, a parallax barrier or lenticular lens type autostereoscopic display (see Patent Documents 2 and 3), and displays three-dimensional images or images of a virtual world on the screen. Of course, the display device 101 may also be a general liquid crystal display that does not provide stereoscopic vision. When the user's head moves, the screen of the display device 101 quickly moves out of the user's field of view. In the display system 100 according to the present disclosure, the robot arm 102 can move the position and orientation of the display device 101 to follow the movement of the position and orientation of the user's head, thereby bringing the screen into the user's field of view. Therefore, the display device 101 may be a normal-sized display of approximately several tens of inches.

[0022] The robot arm 102 is, for example, a serial link robot in which multiple links are connected in series by joint axes, and supports the display device 101 at its distal end. At least some of the multiple joint axes constituting the robot arm 102 are active axes that can be driven by a motor or the like. In order to freely change the position and orientation of the screen of the display device 101 at the distal end in response to movements of the position and orientation of the user's head, it is desirable for the robot arm 102 to have a total of six or more degrees of freedom, including translation in the three X, Y, and Z axes and rotation around the three axes. In the basic configuration example shown in FIG. 1, it is assumed that the robot arm 102 is fixed to a mechanical ground.

[0023] The head position measuring unit 103 measures the head position of the user observing the screen of the display device 101. Specifically, the head position measuring unit 103 is a camera for autostereoscopic vision that is integrated with the display device 101 (for example, attached to the front bezel of the display device 101). The head position measuring unit 103 outputs an image of the user's head captured by this camera for autostereoscopic vision to the control unit 104.

[0024] The control unit 104 calculates the position and orientation of the user's head by measuring the three-dimensional positions of the right and left eyes and one other point (for example, a specific facial part such as the nose or mouth) from the image of the user's head captured by the camera for naked-eye stereoscopic vision, which is output from the head position measurement unit 103. Furthermore, if the user's head moves faster than the speed at which the robot arm 102 can follow it, causing a drive delay, the control unit 104 may predict the head position in addition to actual measurement (described later).

[0025] The control unit 104 controls the driving of the robot arm 102 in accordance with the position of the user's head, and controls the display information of the display device 101 in accordance with the position of the user's head.

[0026] First, the drive control of the robot arm 102 by the control unit 104 will be described. Based on the measurement results by the head position measurement unit 103, the control unit 104 controls the drive of the robot arm 102 so that the screen of the display device 101 can be observed from the user's viewpoint. More specifically, the control unit 104 identifies the user's current viewpoint position based on the calculated position and orientation of the user's head, and then calculates the optimal position and orientation of the display device 101 so that the screen of the display device 101 matches the user's field of view. The position and orientation of the display device 101 correspond to the position and orientation of the end effector of the robot arm 102. The control unit 104 then calculates, by inverse kinematics calculation, the rotation angles of each joint axis for positioning the end effector of the robot arm 102 at a desired position and orientation, generates a drive command for the robot arm 102, and controls the drive of the robot arm 102.

[0027] 2 to 4 each illustrate an example of how the robot arm 102 operates in accordance with the position of the user's head. The robot arm 102 moves the display device 101 at its distal end so that the normal direction of the screen of the display device 101 coincides with the user's line of sight. The robot arm 102 also moves the display device 101 so that the screen of the display device 101 is optimally positioned so that it fits perfectly within the user's field of vision. The human eye is located on a nearly flat face, and it is said that the range that can be seen by both eyes simultaneously is 120 degrees to the left and right. Therefore, the optimal position of the display device 101 is the distance at which the screen size is 120 degrees to the left and right from the center of the user's head.

[0028] In the example shown in FIG. 2 , the user's head is facing forward. Therefore, the robot arm 102 is driven to move the position and orientation of the display device 101 at the distal end so that the screen of the display device 101 faces the user's face in the horizontal direction and is at a fixed distance from the user's head. Also, in the example shown in FIG. 3 , the user's head is facing upward. Therefore, the robot arm 102 is driven to move the position and orientation of the display device 101 at the distal end so that the screen of the display device 101 faces down at the user's face and is at a fixed distance from the user's head. Also, in the example shown in FIG. 4 , the user's head is facing downward. Therefore, the robot arm 102 is driven to move the position and orientation of the display device 101 at the distal end so that the screen of the display device 101 faces up at the user's face and is at a fixed distance from the user's head. The control unit 104 controls the drive of the robot arm 102 to move the position and orientation of the display device 101 as shown in each of FIGS. 2 to 4 .

[0029] Next, a description will be given of the control of the display information of the display device 101 by the control unit 104. The control unit 104 switches the screen of the display device 101 to an image that can be seen from the user's current viewpoint, based on the measurement results by the head position measurement unit 103. However, it is not essential for the present disclosure for the control unit 104 to switch the display image in accordance with the movement of the user's head; the display on the screen may be fixed regardless of the movement of the user's head, or the display image may be switched regardless of the movement of the user's head.

[0030] A mechanism for displaying an image that follows the movement of the user's head on the display device 101 will be described with reference to Fig. 5. Here, it is assumed that a portion 502 is cut out from an original image 501 with a wide angle of view, such as an omnidirectional image or 4K, and displayed on the screen of the display device 101 so as to follow the posture of the user's head.

[0031] In FIG. 5, the depth direction of the user's line of sight is z w axis, horizontal direction is y w axis, the vertical direction is x w axis, and the user's reference axis x w y w z w The origin of the roll θ is the user's viewpoint. z is the z axis of the user's head w Movement around the axis, tilt θ y is the y of the user's head w Movement around the axis, pan θ x is the x in the user's head w This corresponds to the movement around the axis.

[0032] The head position measurement unit 103 measures the displacement (x w , y w , z w ) and rotation around each axis of pan, tilt, and roll (θ z , θ y , θ z ) and outputs the position and attitude information to the control unit 104.

[0033] The control unit 104 receives, for example, posture information (θ z , θ y , θ z ), the display area 502 is moved so as to cancel out the movement of the user's head detected by the head position measurement unit 103. For example, the area 502-1 is rotated in accordance with the roll component of the user's head movement, the area 502-2 is moved in accordance with the tilt component of the user's head movement, and the area 502-3 is moved in accordance with the pan component of the user's head movement. Although not shown in FIG. 5, the displacement (x w , y w , zw As a result, an image 502 can be presented on the screen of the display device 101, in which the display area moves within the original image 501 so as to follow the movement of the user's head.

[0034] The control unit 104 may be physically integrated with the control unit of the robot arm 102. Alternatively, the control unit 104 may be physically integrated with the head position measurement unit 103. Furthermore, the display device 101 and the head position measurement unit 103 may be physically integrated.

[0035] FIG. 6 shows, in the form of a flowchart, the operation of the display system 100 for dynamically and adaptively changing the position of the display device 101 and the screen display in response to the movement of the user's head.

[0036] While the display device 101 is displaying on the screen, the head position measurement unit 103 constantly monitors the position of the user's head and outputs the measurement results to the control unit 104 (step S601). Specifically, the head position measurement unit 103 outputs an image of the user's head captured by the naked eye stereoscopic camera to the control unit 104.

[0037] The control unit 104 calculates the position and orientation of the user's head based on the measurement results of the head position measurement unit 103 (step S602). Specifically, the control unit 104 measures the three-dimensional positions of the right eye, the left eye, and one other point (for example, a specific facial feature such as the nose or mouth) from the image of the user's head captured by the camera for naked-eye stereoscopic vision and output from the head position measurement unit 103, and calculates the position and orientation of the user's head.

[0038] Next, the control unit 104 calculates the optimal position and orientation of the display device 101 so that the normal direction of the screen of the display device 101 coincides with the user's line of sight and the screen is positioned exactly within the user's field of vision (distance from the user's head) (step S603).

[0039] The position and orientation of the display device 101 correspond to the position and orientation of the end effector of the robot arm 102. Therefore, the control unit 104 calculates, by inverse kinematics calculation, the rotation angles of each joint axis for positioning the end effector of the robot arm 102 at a desired position and orientation, from the optimal position and orientation of the display device 101 calculated in step S602, generates a drive command for the robot arm 102 (step S604), and controls the drive of the robot arm 102 (step S605).

[0040] In addition, the control unit 104 calculates the user's viewpoint position in three-dimensional space or virtual space based on the position and posture of the user's head calculated in step S602 (step S611), and cuts out an area corresponding to changes in the position and posture of the user's head from the omnidirectional image or the original image with a wide angle of view, and displays it on the screen of the display device 101 (step S612).

[0041] While the service of the display system 100 continues (No in step S606), the process returns to step S601, and drive control of the robot arm 102 and display control of the display device 101 are repeatedly executed in accordance with the position and posture of the user's head.

[0042] According to the display system 100 of the present disclosure, a display device 101 consisting of a naked-eye stereoscopic display can be used to present three-dimensional images and virtual world images that correspond to the position and posture of the user's head. Because the user's field of vision is not completely obstructed, the user can freely move between real space and virtual space even while viewing images. Furthermore, because the display device 101 can be moved using the robot arm 102 in response to changes in the position and posture of the user's head, the user's observation viewpoint is not limited by the screen size, even if the display device 101 itself is only several tens of inches in size.

[0043] C. Other Configuration Examples and Operation Figure 7 shows another configuration example of a display system according to the present disclosure. The display system 700 shown in Figure 7 includes a display device 701, a robot arm 702, a head position measurement unit 703, a control unit 704, and a moving device 705. The main difference from the display system 100 shown in Figure 1 is that the robot arm 702 is mounted on the moving device 705. Note that in the example shown in Figure 7, the control unit 704 is built into the housing of the moving device 705, but the control unit 704 may be located elsewhere.

[0044] The display device 701 is, for example, a parallax barrier or lenticular lens type autostereoscopic display, and displays three-dimensional images or images of a virtual world on the screen. The display device 701 may be a normal-sized display of about several tens of inches. Of course, the display device 701 may also be a general liquid crystal display that does not support stereoscopic vision.

[0045] The robot arm 702 is, for example, a serial link robot in which multiple links are connected in series via joint axes, and supports a display device 701 at its distal end. The robot arm 702 preferably has a total of six or more degrees of freedom, including translation in the three X, Y, and Z axes and rotation around the three axes. The moving device 705 is equipped with a moving means such as wheels or legs (in the example shown in FIG. 7 , wheels are attached to the four corners of the bottom surface of the moving device 705), and can move the robot arm 702 mounted on the top surface of the device body from its current position. If the moving device 705 is wheeled, it can move stably with less risk of tipping over. Furthermore, if the moving device 705 is legged, it can move easily even on uneven ground, making it more adaptable to the work environment.

[0046] The head position measuring unit 703 measures the head position of the user observing the screen of the display device 701. Specifically, the head position measuring unit 703 is a camera for autostereoscopic vision that is integrated with the display device 701 (for example, attached to the front bezel of the display device 701). The head position measuring unit 703 outputs an image of the user's head captured by this camera for autostereoscopic vision to the control unit 704.

[0047] The control unit 704 calculates the position and posture of the user's head by measuring the three-dimensional positions of the right and left eyes and one other point (for example, a specific facial feature such as the nose or mouth) from an image of the user's head captured by a camera for naked-eye stereoscopic vision and output from the head position measurement unit 703. The control unit 704 calculates changes in the head position at the same location due to physical movements such as the user turning their head or bending over, and also calculates movements in the head position due to the user moving to another location by walking or other movements. Furthermore, if the user's head moves faster than the speed at which the robot arm 702 can track it, causing a drive delay, the control unit 704 may predict the head position in addition to actual measurements (described below).

[0048] The control unit 704 controls the driving of the robot arm 702 and the moving device 705 in accordance with the position of the user's head, and controls the display information of the display device 701 in accordance with the position of the user's head.

[0049] First, the drive control of the robot arm 702 and the moving device 705 by the control unit 704 will be described. The control unit 704 identifies the user's current viewpoint position based on the calculated position and orientation of the user's head, and then calculates the optimal position and orientation of the display device 701 so that the screen of the display device 701 matches the user's field of view. The optimal position of the display device 701 changes in response to a position to which the user has moved, such as by walking, and the user's physical movement at that position (standing upright, bending forward, etc.). The orientation of the display device 701 also changes in response to a head movement such as the user shaking their head (facing forward, looking up, looking down, looking to the side, etc.). The position and orientation of the display device 701 correspond to the current position of the robot arm 702 and the position and orientation of its end effector. Therefore, the control unit 704 calculates the movement path of the moving device 705 and also calculates the rotation angle of each joint axis for positioning the end effector of the robot arm 702 at the desired position and orientation using inverse kinematics calculations, generates movement commands to the moving device 705 and drive commands to the robot arm 702, and controls the movement operation of the moving device 705 and the drive of the robot arm 702.

[0050] 8 to 10 each illustrate an example of how the robot arm 702 operates in accordance with the head position of the user. In each of the examples shown in Fig. 8 to 10, the moving device 705 moves the robot arm 702 in accordance with the user's movement so that the screen of the display device 701 is at a predetermined distance from the user's face. The robot arm 702 then moves the display device 701 at its distal end so that the normal direction of the screen of the display device 701 coincides with the line of sight of the user.

[0051] In the example shown in Figure 8, the user's face is facing forward, but the moving device 705 moves and the robot arm 702 is driven to move the position and orientation of the display device 701 at the distal end so that the screen of the display device 701 is horizontally facing the user's face and at a certain distance from the user's head.

[0052] In the example shown in Figure 9, the user's face is facing upward, but the moving device 705 moves and the robot arm 702 is driven to move the position and orientation of the display device 701 at the distal end so that the screen of the display device 701 is facing down at the user's face and at a certain distance from the user's head.

[0053] 10 , the user's face is facing downward, but the moving device 705 moves and the robot arm 702 is driven to move the position and orientation of the display device 701 at the distal end so that the screen of the display device 701 is in a direction that allows the user to look up at the user's face and is at a certain distance from the user's head. In the example shown in Fig. 10 , the calculated optimal position of the display device 701 is too low, and cannot be achieved by the operation of the robot arm 702 alone if the moving device 705 remains in its original position. Therefore, the optimal position and orientation of the display device 701 are achieved by combining the operation of the robot arm 702 with the operation of moving the moving device 705 backward.

[0054] In this way, the control unit 704 controls the movement of the moving device 705 and the driving of the robot arm 702 in order to change the position and orientation of the display device 701 as shown in each of FIGS.

[0055] Next, we will explain how the control unit 704 controls the display information on the display device 701. The control unit 704 switches the screen of the display device 701 to an image that can be seen from the user's current viewpoint, based on the amount of movement of the moving device 705 and the measurement results from the head position measurement unit 703. Here, we will explain assuming that a portion of the area 502 is cut out from an omnidirectional image or a wide-angle original image 501 and displayed on the screen of the display device 701, as shown in Figure 5.

[0056] A mechanism for displaying an image that follows the movement of the user's head on the display device 701 will be described with reference to Fig. 5. The head position measurement unit 703 measures the displacement (x w , y w , z w ) and rotation around each axis of pan, tilt, and roll (θ z , θ y , θ z ) and outputs it to the control unit 704. The head position measurement unit 703 detects the position and orientation information of the user's head (x w , y w , z w ) includes the amount of displacement due to movement (walking, etc.) in each of the pan, tilt, and roll directions. z , θ y , θ z ), the display area 502 is moved so as to cancel out the movement of the user's head detected by the head position measurement unit 103. For example, the area 502-1 is rotated in accordance with the roll component of the user's head movement, the area 502-2 is moved in accordance with the tilt component of the user's head movement, and the area 502-3 is moved in accordance with the pan component of the user's head movement. w , y w , z w As a result, it is possible to present on the screen of the display device 701 an image 502 whose display area moves within the original image 501 so as to follow the user's movement and head movement.

[0057] FIG. 11 shows, in the form of a flowchart, the operation of the display system 700 for dynamically and adaptively changing the position of the display device 701 and the screen display in response to the user's standing position and head movement.

[0058] While the display device 701 is displaying on the screen, the head position measurement unit 703 constantly monitors the position and posture of the user's head, and outputs information on the position and posture of the user's head to the control unit 704 (step S1101). Specifically, the head position measurement unit 703 outputs an image of the user's head captured by the naked eye stereoscopic camera to the control unit 704.

[0059] The control unit 704 calculates the position and orientation of the user's head based on the measurement results of the head position measurement unit 703 (step S1102). Specifically, the control unit 704 measures the three-dimensional positions of the right eye, the left eye, and one other point (for example, a specific facial feature such as the nose or mouth) from the image of the user's head captured by the camera for naked-eye stereoscopic vision and output from the head position measurement unit 703, and calculates the position and orientation of the user's head.

[0060] Next, the control unit 704 calculates the optimal position and orientation of the display device 701 so that the normal direction of the screen of the display device 701 coincides with the user's line of sight and the screen is positioned exactly within the user's field of vision (distance from the user's head) (step S1103).

[0061] The position and orientation of the display device 701 correspond to the position and orientation of the end effector of the robot arm 702. The control unit 704 checks whether the optimal position of the display device 701 calculated in step S1102 is within the movable range of the end effector at the current position of the robot arm 702 (step S1104).

[0062] If the optimal position of the display device 701 calculated in step S1103 exceeds the movable range of the end effector of the robot arm 702 (No in step S1104), the control unit 704 determines a movement path for the moving device 705 to bring the optimal position of the display device 701 into the movable range of the end effector of the robot arm 702 (step S1105), and controls the movement of the moving device 705 (step S1106).

[0063] Next, the control unit 704 calculates the rotation angles of each joint axis for positioning the end effector of the robot arm 702 at the desired position and orientation using inverse kinematics calculations from the optimal position (or the position after movement in step S1106) and orientation of the display device 101 calculated in step S1103, generates a drive command for the robot arm 102 (step S1107), and controls the drive of the robot arm 102 (step S1108).

[0064] In addition, the control unit 704 calculates the user's viewpoint position in three-dimensional space or virtual space based on the position and posture of the user's head calculated in step S1102 (step S1111), cuts out an area corresponding to changes in the user's position and posture from the omnidirectional image or the wide-angle original image, and displays it on the display device 101 (step S1112).

[0065] While the service of the display system 700 continues (No in step S1109), the process returns to step S1101, and drive control of the robot arm 702 and display control of the display device 701 are repeatedly executed in accordance with the position and posture of the user's head.

[0066] According to the display system 700 of the present disclosure, a display device 701 consisting of a naked-eye stereoscopic display can be used to present three-dimensional images and virtual world images that correspond to the user's movement and the position and posture of their head. Because the user's field of vision is not completely obstructed, they can freely move between real space and virtual space even while viewing images. Furthermore, since the robot arm 702 can be moved in accordance with the user's movement using the movement device 705, and the robot arm 702 can be used to move the display device 701 in accordance with changes in the position and posture of the user's head, the viewpoint from which the user can observe is not limited by the screen size, even if the display device 701 itself is only a few tens of inches in size.

[0067] D. Calculation and Prediction of Head Position This section D describes a method for calculating the optimal position and orientation of the display device 101 in the display system 100 by tracking the movement of the user's head, and a method for predicting the user's head position. For example, if the speed of the user's head movement is too fast compared to the response of the robot arm 102, the predicted result can be used instead of actually measuring the user's head position. A similar method can be applied to the display system 700.

[0068] D-1. Calculation of the position of the display device that tracks head movement Point group a, which is at a fixed position relative to the display coordinate system of the display device 101, is shown in the following equation (1). Also, assume that point group a moves to point group b by rotational movement R and translational movement T. Point group b is shown in the following equation (2). The following equation (3) mathematically shows that point group a moves to point group b by rotational movement R and translational movement T. Point groups a and b are three-dimensional. Furthermore, R is a rotation matrix that represents the rotational movement, and T is a translation matrix that represents the translational movement. However, point groups a and point groups b each have three or more points (that is, n is an integer greater than or equal to 3). This is because three points are required to calculate a three-dimensional position.

[0069]

[0070]

[0071]

[0072] Both the rotational motion R and the translational motion T are unknown. A method for determining the estimated value R^ of R and the estimated value T^ of T will be described below. Note that the estimated value of variable X is expressed in mathematical expressions by placing an accent symbol "^ (hat)" above the letter "X", but in this specification, the estimated value of variable X will be expressed as "X^", which is the letter "X" with an accent symbol "^ (hat)" immediately after it.

[0073] First, the center of gravity of point group a and the center of gravity of point group b are found using the following formulas (4) and (5), respectively. The center of gravity of point group Y is expressed in formulas by adding an accent symbol "- (superscript bar)" above the letter "Y", but in this specification, the center of gravity of point group Y is expressed as "Y-", which is the letter "Y" with the accent symbol "-" immediately after it.

[0074]

[0075]

[0076] Next, using the following formula (6), the deviation m between the point group a and its center of gravity a- is calculated. a , and the deviation m between the point group b and its center of gravity b b Ask for.

[0077]

[0078] Here, we want to find the rotation matrix R given by the following equation (7).

[0079]

[0080] For the correlation matrix K expressed by the following equation (8), an estimated value R^ of the rotation matrix is ​​calculated so as to satisfy the following equation (9). r [.] is a trace.

[0081]

[0082]

[0083] To obtain the estimated value R^ of the rotation matrix, the correlation matrix K is subjected to singular value decomposition (SVD) as shown in the following equation (10), and the estimated value R^ of the rotation matrix can be obtained as the following equation (11), where det[.] is a determinant.

[0084]

[0085]

[0086] By substituting the estimated value R^ of the rotation matrix into the above equation (3), the estimated value T^ of the translation matrix can also be obtained as shown in the following equation (12).

[0087]

[0088] Point clouds a and b correspond to three or more observation points before and after the user's head performs a rotational movement R and a translational movement T. In display system 100, head position measurement unit 103 can measure the three-dimensional positions of point clouds a and b based on images captured by a camera for naked-eye stereoscopic vision. Control unit 104 can use the above equations (1) to (12) to respectively obtain an estimated value R^ of the rotation matrix and an estimated value T^ of the translation matrix.

[0089] The control unit 104 can calculate the movement amounts of the joint angles of the robot arm 102 based on an inverse kinematics operation or the like from the rotation matrix estimate R^ and the translation matrix estimate T^ to move the display device 101 to an optimal position and orientation in accordance with the rotational and translational movements of the user's head. The control unit 104 can then drive each joint of the robot arm 102 based on the calculation results, thereby moving the display device 101 to an optimal position and orientation in which the screen is always directly in front of the user's face. Furthermore, the control unit 104 controls the screen of the display device 101 to display an image that follows the rotational and translational movements of the user's head, based on the rotation matrix estimate R^ and the translation matrix estimate T^.

[0090] FIG. 12 shows the calculation of the position of the display device following the user's head movement. In FIG. 12, the point group a(=(a1 T , a2 T , a3 T )) and the position and orientation Face_A of the user's face obtained from this point group a. The position and orientation Display_A of the display device 101 that is optimal for the position and orientation Face_A of the user's face obtained from this point group a. The point group b (=(b T , b2 T , b3 T ) and the position and orientation Display_B of the display device 101 that are optimal for the position and orientation Face_B of the user's face determined from the point group b. Display_A is the position and orientation of the display device 101 that is in front of the user's face at the position and orientation Face_A defined by the point group a. Also, Display_B is the position and orientation of the display device 101 that is in front of the user's face at the position and orientation Face_B defined by the point group b.

[0091] The head position measurement unit 103 can measure the three-dimensional position of each observation point (a1, a2, a3) included in point cloud a and the three-dimensional position of each observation point (b1, b2, b3) included in point cloud b based on images of the user's head captured by a camera for naked-eye stereoscopic vision. In the example shown in Figure 12, the observation points of point cloud a and point cloud b include the right eye, left eye, and one other point, the nose. Point cloud a corresponds to the position and orientation Face_A of the user's face before the user's head performs the rotational movement R and translational movement T. Point cloud b corresponds to the position and orientation Face_B of the user's face after the user's head performs the rotational movement R and translational movement T.

[0092] The control unit 104 can use the above equations (1) to (12) to calculate estimated values ​​R^ and T^ representing the rotational movement R and translational movement T of the user's head, respectively, from the measurement results of the three-dimensional positions of point group a and point group b. Therefore, the control unit 104 can calculate the amount of movement of each joint of the robot arm 102, which is required to move the display device 101 from Display_A to Display_B in accordance with the rotational movement and translational movement of the user's head, based on inverse kinematics calculations or the like, from the estimated values ​​R^ and T^ of each matrix. The control unit 104 can then drive each joint of the robot arm 102 based on the calculation results to actually move the display device 101 from Display_A to Display_B. In this way, the control unit 104 can estimate the movement of the user's face and move the entire display device 101 in accordance with the amount of face movement.

[0093] D-2. Predicting Head Position As described in Section D-1 above, when the position and posture of the user's face are displaced, the control unit 104 can calculate movement parameters based on the measurement results by the head position measurement unit 103 so that the screen of the display device 101 is positioned directly in front of the user's face, and drive the robot arm 102.

[0094] However, due to a drive delay of the robot arm 102, the position and posture of the user's face may have already been further displaced by the time the movement of the robot arm 102 is actually completed. The drive delay is caused by, for example, a slow drive speed of the robot arm 102 or an excessively fast movement speed of the user's head.

[0095] Therefore, the head position measurement unit 103 may not only actually measure the movement of the user's head, but also predict the future head position based on past head movement. Therefore, the control unit 104 can calculate movement parameters based on the measurement results by the head position measurement unit 103 so that the screen of the display device 101 is positioned directly in front of the predicted user's face, and drive the robot arm 102. Even if a drive delay occurs in the robot arm 102, the movement of the robot arm 102 will already be completed by the time the position and posture of the user's face are displaced. Therefore, the screen of the display device 101 is always positioned directly in front of the user's face.

[0096] The head position measurement unit 103 may be configured to constantly predict the position of the user's head. Alternatively, the head position measurement unit 103 may be configured to constantly predict the position of the user's head only when a drive delay occurs in the robot arm 102 (or when a drive delay is detected).

[0097] There are no particular limitations on the algorithm used by the head position measurement unit 103 to predict the position of the user's head. For example, the head position measurement unit 103 may predict the position of the user's head using a neural network model that has previously learned about the movement of the user's head.

[0098] 13 shows a position prediction of the display device that follows the movement of the user's head. In FIG. 13, the three-dimensional position of point cloud a observed before the user's head performs rotational movement R and translational movement T and the position and orientation Display_A of the display device 101 that are optimal for the position and orientation Face_A of the user's face obtained from this point cloud a, the three-dimensional position of point cloud b observed after the user's head performs rotational movement R and translational movement T and the position and orientation Display_B of the display device 101 that are optimal for the position and orientation Face_B of the user's face obtained from this point cloud b, and a predicted point cloud c (=(c1 T , c2 T , c3 T) and the position and orientation Future_Display of the display device 101 that are optimal for the future position and orientation Future_Face of the user's face determined from this point cloud c. Future_Display is the position and orientation of the display device 101 that is in front of the user's face at the position and orientation Future_Face defined by the predicted point cloud c.

[0099] The head position measurement unit 103 can predict the three-dimensional position of each observation point (c1, c2, c3) of the future point cloud c using a trained neural network model from the actually measured three-dimensional positions of each observation point of the point clouds a and b (where the observation points include the right eye, left eye, and one other point, the nose). The control unit 104 can obtain estimated values ​​R^ and T^ of each matrix representing the predicted rotational movement R and translational movement T of the user's head, respectively, from the three-dimensional position information of the point clouds a to c, and can calculate the movement amount of each joint of the robot arm 102 based on inverse kinematics calculations or the like from the estimated values ​​R^ and T^ of each matrix, in order to move the display device 101 to Future_Display in accordance with the rotational movement and translational movement of the user's head. Then, the control unit 104 can drive each joint of the robot arm 102 based on the calculation result to actually move the display device 101 to the position and orientation of Future_Display. In this way, the control unit 104 can predict the movement of the user's face and realize the movement of the entire display device 101 according to the predicted amount of face movement without causing a drive delay.

[0100] E. Application Examples This section E mainly describes application examples of the display system 700 shown in Fig. 7. Unlike the display system 100 shown in Fig. 1, the display system 700 shown in Fig. 7 is a mobile system in which a robot arm 700 supporting a display device 701 is mounted on a mobile device 705.

[0101] E-1. Searching for a Virtual Space This section E-1 describes an application example in which the display system 700 is applied to searching for a virtual space. In the display system 700, the moving device 705 moves so that the screen of the display device 701 is in front of the user's face and at a certain distance from the user's head. Here, it is assumed that a partial area is cut out from an omnidirectional image of the virtual space and displayed on the screen of the display device 701. Based on the amount of movement of the moving device 705 and the measurement results by the head position measurement unit 703, the control unit 704 switches the screen of the display device 701 to an image that is visible from the user's current viewpoint.

[0102] 14 shows a situation in which a user is exploring a virtual space (for example, the interior of a virtual house). As the user moves forward, the mobile device 705 moves backward, adjusting the user's viewing angle (into the virtual space) to be constant (i.e., so that the screen of the display device 701 is at a constant distance from the user's head). The control unit 704 sequentially switches the screen of the display device 701 to an image that can be seen from the user's current viewpoint, based on the amount of movement of the mobile device 705 and the measurement results by the head position measurement unit 703.

[0103] For example, a screen display when the virtual space is the interior of a virtual house and the user is walking around the house will be described. Fig. 15 shows the display screen of the display device 701 when the user is walking down a hallway 1501. When the user advances down the hallway 1501, reaches a door 1502 ahead, and opens the door 1502 to enter a room, the screen of the display device 701 switches to a view of the interior of the room beyond the door 1502, as shown in Fig. 16.

[0104] Although not shown, when the user looks up at any location in the virtual space, the robot arm 702 is driven to move the position and orientation of the display device 701 at the distal end so that the screen of the display device 701 is directed downward at the user's face and at a certain distance from the user's head, and the screen of the display device 701 switches to an image that the user sees in the direction looking up (for example, of the ceiling). Similarly, when the user looks down at any location in the virtual space, the robot arm 702 is driven to move the position and orientation of the display device 701 at the distal end so that the screen of the display device 701 is directed upward at the user's face and at a certain distance from the user's head, and the screen of the display device 701 switches to an image that the user sees in the direction looking down (for example, of the floor).

[0105] E-2. Application to Teleconferences This section E-2 describes an application example in which the display system 700 is applied to a teleconference. In this case, a display system 700 is installed at each location of the teleconference. If there are multiple conference participants at one location, multiple display systems 700 may be installed at that location (e.g., one for each conference participant). Of course, there may be a location where no display system 700 is installed.

[0106] Fig. 17 shows an example in which display system 700 is applied to a remote conference. For simplicity of explanation, Fig. 17 assumes a remote conference system 1600 connecting two locations, with display systems 700L and 700R located at locations L and R on the left and right sides of the figure, respectively.

[0107] For example, an image of conference participant L captured by a camera for autostereoscopic vision serving as head position measurement unit 703L in display system 700L at one location L is transferred to the other location R and displayed on the screen of display device 701R on the display system 700R side. Similarly, an image of conference participant R captured by a camera for autostereoscopic vision serving as head position measurement unit 703R in display system 700R is transferred and displayed on the screen of display device 701L on the display system 700L side. Of course, instead of using the camera of head position measurement unit 703, a dedicated camera for capturing images of conference participants may be separately installed at each of locations L and R.

[0108] At one location L, a conference participant L may move forward to approach the conference participant R, or conversely, move backward to move away from the conference participant R. In such a case, in the display system 700L, when the conference participant L moves forward, the mobile device 705L moves backward (see FIG. 18 ), and conversely, when the conference participant L moves backward, the mobile device 705L moves forward (see FIG. 19 ), thereby adjusting the viewing angle of the conference participant L to be constant (i.e., so that the screen of the display device 701L is at a constant distance from the head of the conference participant L). Furthermore, the control unit 704L performs a zoom process (not shown) on the image displayed on the screen of the display device 701L in accordance with the virtual distance between the conference participant L and the other conference participant R resulting from the conference participant L's movement (forward or backward).

[0109] Furthermore, at one location L, a conference participant L can turn his / her face downward and look into the other location. In such a case, as shown in Fig. 10 , in the display system 700L, the robot arm 702L is driven to move the position and orientation of the display device 701L at the distal end so that the screen of the display device 701L is oriented in a direction that allows the conference participant L to look up at the face of the conference participant L and is at a certain distance from the head of the conference participant L. Furthermore, the control unit 704L switches the display on the screen of the display device 701L to an image of the conference participant L looking into the other location (for example, an image of the floor).

[0110] The above describes the operation of the display system 700L in response to the behavior of the conference participant L at one location L, but the display system 700R can also operate in response to the behavior of the conference participant R at the other location R.

[0111] Remote conferences using information terminals such as personal computers, tablets, and smartphones as conference terminals are already widely practiced. By using the display system 700 according to the present disclosure as a conference terminal, conference participants can be provided with stereoscopic images of other locations that correspond to their own movements, enabling more realistic and immersive remote conferences. For example, when remote conference participants shake hands, a realistic and immersive image of the other party can be displayed as they move closer to each other through the screen. It is also possible to hold remote conferences using the display system 100 (see FIG. 1 ) as a conference terminal, even if it does not include a means of transportation.

[0112] E-3. Pushing Operation by User The display system 700 equipped with the moving device 705 can move autonomously and actively in response to the user's moving actions such as walking, and can also move passively by the user's pushing action, etc.

[0113] 20 shows a state in which a handle 705A is attached to the front of the device body of the moving device 705, and a user uses this handle 705A to push the moving device 705 forward like a handcart. The entire display system 700 moves forward as the moving device 705A moves. The moving device 705 may have a storage space for storing luggage within the device body, or may be equipped with a rack (not shown) for placing luggage. When the moving device 705 is being pushed by the user as shown in FIG. 20, it is desirable to keep the wheels of the moving device 705 in a free-rotating state.

[0114] 21 also shows a state in which a handle 701A is attached to the front of the display device 701, and a user is pushing the handle 701A forward. The force of the user pushing the handle 701A is transmitted to the moving device 705A via the robot arm 702, and the entire display system 700 moves forward. When performing a moving operation by the user pushing with their hand as shown in FIG. 21, it is desirable to fix each joint of the robot arm 702 and to set the wheels of the moving device 705 in a free-rotating state so that the pushing force of the user can be easily transmitted to the moving device 705.

[0115] 20 and 21 , when the moving device 705 is moved by a user's hand push operation, the amount of movement and direction of movement of the moving device 705 (or the display device 701, or the entire display system 700) can be calculated based on the amount of rotation and rotation speed of each wheel. Then, even while the user is pushing the moving device 705 as shown in Fig. 20 or 21 , the control unit 704 controls the drive of the robot arm 702 in accordance with the position of the user's head, and controls the display information of the display device 701 in accordance with the position of the user's head.

[0116] First, the drive control of the robot arm 702 and the moving device 705 will be described. The control unit 704 identifies the user's current viewpoint position based on the amount and direction of movement of the moving device 705 calculated from the amount and speed of rotation of each wheel and the measurement results of the three-dimensional positions of multiple observation points of the user's head by the head position measurement unit 703, and then calculates the position and orientation of the display device 701 so that the screen of the display device 701 matches the user's field of view. The control unit 704 then calculates, by inverse kinematics calculation, the rotation angles of each joint axis of the robot arm 702 to position the display device 701 at a desired position and orientation, generates a drive command for the robot arm 702, and controls the drive of the robot arm 702.

[0117] Next, we will explain the control of the display information of the display device 701 by the control unit 704. Based on the amount and direction of movement of the moving device 705 calculated from the amount of rotation and rotation speed of each wheel and the measurement results by the head position measurement unit 703, the control unit 704 switches the screen of the display device 701 to an image that can be seen from the current viewpoint of the user after performing the push operation.

[0118] F. Configuration of Information Processing Device In this section F, an information processing device that can be used as the control unit 104 of the display system 100 shown in FIG. 1 and the control unit 704 of the display system 700 shown in FIG. 7 will be described.

[0119] 22 shows an example of the hardware configuration of an information processing device 2000. This information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013.

[0120] The CPU 2001 controls the overall operation of the information processing device 2000 in accordance with various programs. When performing computationally intensive processing such as neural network model learning on the information processing device 2000, it is desirable that the CPU 2001 be a multi-core CPU (e.g., Apple M1 Max, etc.), or that the information processing device 2000 further be equipped with a multi-core processor (e.g., NVIDIA R6000, etc.) such as a GPU (Graphics Processing Unit) or GPGPU (General-purpose computing on graphics processing units) in addition to the CPU 2001. However, hereinafter, for convenience, these will be collectively referred to simply as the CPU 2001.

[0121] The ROM 2002 stores in a nonvolatile manner programs (such as a basic input / output system) and calculation parameters used by the CPU 2001. The RAM 2003 is used to load programs used in the execution of the CPU 2001 and to temporarily store parameters such as work data that change as appropriate during program execution. Programs loaded into the RAM 2003 and executed by the CPU 2001 include, for example, various application programs and an operating system (OS).

[0122] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004, which includes a CPU bus and the like. The CPU 2001 executes various application programs in an execution environment provided by an OS through the cooperative operation of the ROM 2002 and RAM 2003, thereby realizing various functions and services. If the information processing device 2000 is a personal computer, the OS may be, for example, Microsoft Windows (registered trademark) or Unix (registered trademark). Examples of application programs include the following. At least some of the application programs may be computer programs provided as libraries. (1) Head position estimation / prediction program: A program for estimating or predicting the position and orientation of the user's head based on the measurement results of the head position measurement unit 103 or 703. (2) Display device position / orientation calculation program: A program for calculating the optimal position and orientation of the display device 101 or 701 corresponding to the user's head position and orientation estimated or predicted using the head position estimation / prediction program. The display device position and orientation calculation program may be a program that predicts the position of the user's head using a trained neural network model. (3) Inverse Kinematics Calculation Program: A program that calculates, by inverse kinematics calculation, the joint angles of the robot arm 102 or 702 to move the display device 101 or 701 (end effector) to an optimal position and orientation. The inverse kinematics calculation program has a function for calculating the angle of each joint axis of the robot arm 102 or 702 to achieve the desired position and orientation of the display device 101 or 701. For example, the inverse kinematics calculation program has a function for inputting a rotation matrix R and a translation matrix T that respectively represent changes in the position and orientation of the display device 101 or 701 and calculating the angle of each joint axis of the robot arm 102 or 702. (4) Drive Control Program: A program that generates drive commands for the robot arm 102 or 702 based on the joint angles calculated using the inverse kinematics calculation program and controls the drive of the robot arm 102 or 702.

[0123] The host bus 2004 is connected to an expansion bus 2006 via a bridge 2005. The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or PCI Express, and the bridge 2005 is based on the PCI standard. However, the information processing device 2000 does not need to be configured so that the circuit components are separated by the host bus 2004, bridge 2005, and expansion bus 2006, and may be implemented so that almost all circuit components are interconnected by a single bus (not shown).

[0124] The interface unit 2007 connects peripheral devices such as an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013 in accordance with the standards of the expansion bus 2006. However, not all of the peripheral devices shown in Fig. 22 are necessarily required, and the information processing device 2000 may further include peripheral devices not shown. Furthermore, the peripheral devices may be built into the main body of the information processing device 2000, or some of the peripheral devices may be externally connected to the main body of the information processing device 2000.

[0125] The input unit 2008 is composed of an input control circuit and the like that generates an input signal based on an input from a user and outputs the signal to the CPU 2001. When the information processing device 2000 is a personal computer, the input unit 2008 may include a keyboard, a mouse, a touch panel, and may further include a camera and a microphone. The input unit 2008 may include a camera for naked-eye stereoscopic vision used in the head position measurement unit 103 or 703, or may include an interface that inputs a signal from the head position measurement unit 103 or 703.

[0126] The output unit 2009 includes, for example, a display device such as a liquid crystal display (LCD) device, an organic electroluminescence (EL) display device, or an LED (light emitting diode), as well as an audio output device such as a speaker. The display device referred to here may include the display device 101 or 701 (naked-eye stereoscopic display) that is a component of the display system 100 or 700. The output unit 2009 may also include a monitor display attached to the main body of the mobile device 705, which displays the status of the display system 700.

[0127] The storage unit 2010 stores files such as programs (applications, OS, etc.) executed by the CPU 2001 and various data. The storage unit 2010 is configured with a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may also include an external storage device.

[0128] The removable storage medium 2012 is a storage medium configured as a cartridge, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 113. The drive 2011 outputs data read from the removable storage medium 2012 to the RAM 2003 or the storage unit 2010, and writes data on the RAM 2003 or the storage unit 2010 to the removable storage medium 2012.

[0129] The communication unit 2013 is a device that performs wireless communication such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or a cellular communication network such as 4G or 5G. The communication unit 2013 may also include terminals such as a Universal Serial Bus (USB) or a High-Definition Multimedia Interface (HDMI) (registered trademark), and may further include a function for performing HDMI (registered trademark) communication with a USB device such as a scanner or printer, or a display.

[0130] The information processing device 2000 is configured by, for example, an information terminal such as a personal computer, a tablet, or a smartphone. The information processing device may be mounted on the main body of the mobile device 705, or may be connected to any of the components of the display system 100 or 700 by wireless or wired means.

[0131] The present disclosure has been described in detail above with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the spirit of the present disclosure. Furthermore, the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto, and additional effects not described in this specification may exist.

[0132] The display system disclosed in this specification can be used, for example, as a conference terminal located at each location in a remote conference system, and can present three-dimensional images and images of virtual space easily and at low cost, replacing a CAVE system. The gist of this disclosure is not limited to the specific embodiments described in this specification. Furthermore, each component of the display system disclosed in this specification can be individually modified depending on the application and field of application.

[0133] In this specification, an embodiment has been described in which the display device is supported by a robot arm with a serial link structure, but the display device may also be supported by a robot arm with a parallel link structure, or a robot arm consisting of a combination of serial links and parallel links.

[0134] In this specification, the description has been centered on an embodiment in which a display device uses a stereoscopic display with no eyes, but a two-dimensional display may be used instead of a stereoscopic display as long as it is visible to the naked eye.

[0135] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0136] The present disclosure may also be configured as follows.

[0137] (1) A display system comprising: a display device; a robot arm supporting the display device; a measurement unit that measures the head of a user observing a screen of the display device; and a control unit that controls the drive of the robot arm based on the measurement results by the measurement unit so that the screen of the display device can be observed from the viewpoint of the user.

[0138] (2) The display system described in (1) above further includes a mobile device that carries the robot arm, and the control unit controls the movement of the mobile device so that, when the user moves, the screen of the display device is in a position that can be observed from the user's viewpoint.

[0139] (3) The control unit calculates, based on the measurement results by the measurement unit, an optimal position and orientation where the normal direction of the screen of the display device coincides with the line of sight of the user and the screen of the display device fits exactly within the field of view of the user, and calculates the rotation angles of each joint axis of the robot arm using inverse kinematics calculations to position the display device at the position and orientation, and controls the drive of the robot arm. A display system described in any one of (1) or (2) above.

[0140] (4) The display system according to any one of (1) to (3), wherein the display device is a naked-eye stereoscopic display and displays a three-dimensional image or an image of a virtual world.

[0141] (5) The control unit further controls the screen of the display device to display an image that follows the rotational and translational movements of the user's head based on the measurement results of the measurement unit. The display system described in any one of (1) to (4) above.

[0142] (6) The display system described in any one of (1) to (5) above, wherein the measurement unit measures the three-dimensional positions of three or more observation points including the right eye and the left eye from an image of the user's head captured by a camera for naked-eye stereoscopic vision, and the control unit estimates the position and posture of the user's head based on the measurement results of the three-dimensional positions of the observation points by the measurement unit.

[0143] (7) A display system described in any one of (1) to (5) above, wherein the measurement unit measures the three-dimensional positions of three or more observation points including the right eye and the left eye before and after the user's head performs rotational and translational movements, and the control unit calculates an estimated value R^ of the rotational movement of the user's head and an estimated value T^ of the translational movement of the user's head based on the measurement results of the three-dimensional positions of the observation points by the measurement unit.

[0144] (8) The display system described in (7) above, wherein the control unit calculates the position and orientation of the display device that follows the rotational and translational movements of the user's head based on the estimated values ​​R^ and T^, and controls the drive of the robot arm.

[0145] (9) The display system described in (8) above, wherein the control unit controls the screen of the display device to display an image that follows the rotational and translational movements of the user's head based on the estimated values ​​R^ and T^.

[0146] (10) The display system described in any one of (1) to (9) above, wherein the control unit predicts future head movement based on past head movement of the user, calculates the position and posture of the display device that follows the predicted head movement of the user, and controls the drive of the robot arm.

[0147] (11) A control method for a display system configured by supporting a display device with a robot arm, the control method comprising: a measurement step of measuring the head of a user observing a screen of the display device; and a control step of controlling the drive of the robot arm based on the measurement results in the measurement step so that the screen of the display device can be observed from the viewpoint of the user.

[0148] (12) The control method described in (11) above, wherein the display system further includes a moving device that carries the robot arm, and the control step controls the movement of the moving device so that, when the user moves, the screen of the display device is in a position that can be observed from the user's viewpoint.

[0149] (13) The control method according to any one of (11) or (12) above, wherein in the control step, an optimal position and orientation are calculated based on the measurement results in the measurement step, where the normal direction of the screen of the display device coincides with the line of sight of the user and the screen of the display device fits exactly within the field of view of the user, and the rotation angles of each joint axis of the robot arm for positioning the display device at the position and orientation are calculated by inverse kinematics calculation, and the drive of the robot arm is controlled.

[0150] (14) The control method described in any one of (11) to (13) above, further comprising a display control step of controlling the screen of the display device to display an image that follows the rotational and translational movements of the user's head based on the measurement results in the measurement step.

[0151] (15) A control method according to any one of (11) to (14) above, wherein in the measurement step, the three-dimensional positions of three or more observation points including the right eye and the left eye are measured before and after the user's head performs rotational and translational movements; and in the control step, an estimated value R^ of the rotational movement of the user's head and an estimated value T^ of the translational movement are calculated based on the measurement results of the three-dimensional positions of the observation points in the measurement step; and based on the estimated values ​​R^ and T^, the position and posture of the display device that follows the rotational and translational movements of the user's head are calculated to control the drive of the robot arm.

[0152] (16) The control method described in any one of (11) to (15) above, wherein the control step predicts future head movement based on past head movement of the user, calculates the position and posture of the display device that follows the predicted head movement of the user, and controls the drive of the robot arm.

[0153] DESCRIPTION OF SYMBOLS 100...display system, 101...display device, 102...robot arm, 103...head position measurement unit, 104...control unit 700...display system, 701...display device, 701A...handle 702...robot arm, 703...head position measurement unit 704...control unit, 705...movement device, 705A...handle 2000...information processing device, 2001...CPU, 2002...ROM 2003...RAM, 2004...host bus, 2005...bridge 2006...expansion bus, 2007...interface unit 2008...input unit, 2009...output unit, 2010...storage unit 2011...drive, 2012...removable recording medium 2013...communication unit

Claims

1. A display system comprising: a display device; a robot arm supporting the display device; a measurement unit that measures the head of a user observing a screen of the display device; and a control unit that controls the drive of the robot arm based on the measurement results by the measurement unit so that the screen of the display device can be observed from the viewpoint of the user.

2. The display system according to claim 1, further comprising a moving device mounted with the robot arm, wherein the control unit controls the movement of the moving device so that, when the user moves, the screen of the display device is located in a position observable from the viewpoint of the user.

3. The display system of claim 1, wherein the control unit calculates, based on the measurement results by the measurement unit, an optimal position and orientation in which the normal direction of the screen of the display device coincides with the line of sight of the user and the screen of the display device fits exactly within the field of view of the user, and calculates, by inverse kinematics calculations, the rotation angles of each joint axis of the robot arm for positioning the display device at said position and orientation, and controls the drive of the robot arm.

4. The display system according to claim 1, wherein the display device is a naked eye stereoscopic display and displays a three-dimensional image or a virtual world image.

5. The display system according to claim 1, wherein the control unit further controls the screen of the display device to display an image that follows the rotational and translational movements of the user's head based on the measurement results by the measurement unit.

6. The display system of claim 1, wherein the measurement unit measures the three-dimensional positions of three or more observation points including the right eye and left eye from an image of the user's head captured by a camera for naked-eye stereoscopic vision, and the control unit estimates the position and posture of the user's head based on the measurement results of the three-dimensional positions of the observation points by the measurement unit.

7. The display system of claim 1, wherein the measurement unit measures the three-dimensional positions of three or more observation points including the right eye and left eye before and after the user's head performs rotational and translational movements, and the control unit calculates an estimated value R^ of the rotational movement and an estimated value T^ of the translational movement of the user's head based on the measurement results of the three-dimensional positions of the observation points by the measurement unit.

8. The display system according to claim 7, wherein the control unit calculates the position and orientation of the display device that follows the rotational and translational movements of the user's head based on the estimated values ​​R^ and T^, and controls the drive of the robot arm.

9. The display system according to claim 8, wherein the control unit controls the screen of the display device to display an image that follows the rotational and translational movements of the user's head based on the estimated values ​​R^ and T^.

10. The display system of claim 1, wherein the control unit predicts future head movement based on past head movement of the user, calculates the position and orientation of the display device that follows the predicted head movement of the user, and controls the drive of the robot arm.

11. A control method for a display system in which a display device is supported by a robot arm, the control method comprising: a measurement step of measuring the head of a user observing the screen of the display device; and a control step of controlling the drive of the robot arm based on the measurement results in the measurement step so that the screen of the display device can be observed from the viewpoint of the user.

Images