Information processing device and information processing method

Abstract

This information processing device which is used by being worn by a user and this information processing method involve: reproducing a stereoscopic virtual video object; disposing and displaying the reproduced virtual video object so as to be superimposed on a real object; and recording user information, which contains the disposition position where the virtual video object is disposed and / or the position and line-of-sight direction of the user. When it is detected that the user has gone out of a range which is within a prescribed distance from the disposition position, the reproduction of the virtual video object is stopped. When the user has entered said range after stoppage of the reproduction of the virtual video object, the reproduction of the virtual video object is resumed on the basis of the recorded user information.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] In recent years, Augmented Reality (AR) technology has been widely used. AR technology adds digital information to the real world, and virtual objects created in a virtual space using computer graphics (CG) or the like are reflected and augmented in the real space. Information processing devices that utilize this technology, which allow users to easily handle virtual objects while recognizing the real space in three dimensions, are becoming widespread. One example of such a device is a head-mounted display (HMD), which is worn on the head and has a display unit, a camera unit, and the like.

[0003] Patent Document 1 describes the following as a method for pausing or playing the action of a video of a virtual object (hereinafter, sometimes referred to as a video virtual object). That is, Patent Document 1 states that "the playback unit 107 plays back video data stored in the storage unit 104 (301). In parallel with the playback process by the playback unit 107, the mixed reality processing unit 103 acquires the user's viewpoint position (or the user's position) and the direction of their gaze using the position and orientation estimation result by the self-position and orientation estimation unit 106 and the gaze tracking function (S302). The mixed reality processing unit 103 calculates the gaze deviation Φ from the video playback screen G and the distance D between the user's self-position and the video playback screen G. The playback unit 107 adjusts the playback speed in accordance with the deviation Φ and the distance D (S303). For example, if the deviation Φ is a predetermined angle and the distance D is equal to or greater than a predetermined value, the playback unit 107 slows down or stops the playback speed. Note that at least one of the conditions for the deviation Φ and the distance D may be satisfied. This is a criterion for determining whether the user is looking at the video playback screen G."

[0004] Patent Document 2 also states that "the user terminal 1500 has a function of acquiring measurement data that measures the position and attitude of the user terminal 1500, and places a subject object and a virtual camera in a virtual three-dimensional space, and controls the position and attitude of the virtual camera in conjunction with the measurement data. The user terminal 1500 then generates a virtual space image taken by the virtual camera and displays it as a monitor image, while recording camerawork data 550 that can reproduce the position and attitude of the virtual camera." and "the playback image generation control unit 232 controls the generation of a playback image, which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork, based on the camerawork data 550 recorded by the recording control unit 224."

[0005] International Publication No. WO2021 / 132168 A1 JP 2020-119334 A

[0006] As described above, Patent Document 1 discloses that the stopping and playback of a video virtual object screen are controlled based on the position and direction of the user's line of sight and the distance from a placed planar video virtual object. However, no consideration is given to the case of a three-dimensional video virtual object that can be viewed from multiple positions and directions. Therefore, there is a problem in stopping and resuming viewing of a three-dimensional video virtual object in an appropriate state and at an appropriate timing.

[0007] As described above, Patent Document 2 discloses controlling the playback of a video virtual object based on information related to camerawork (camera position and orientation). However, it does not provide any information related to the user's line of sight with respect to the video virtual object or the distance from the user. Therefore, a problem can be considered in that it is not possible to control the stopping and playback of a video virtual object based on the position, directional deviation, or distance of the user's line of sight with respect to the video virtual object. Furthermore, while Patent Document 2 discloses a suitable "video creation" technology, it does not disclose any suitable "viewing" technology, leaving a problem in terms of stopping and resuming "viewing" at a suitable timing. Therefore, in consideration of the above-mentioned problems, the present invention aims to provide an information processing device and an information processing method that enable easy and convenient pausing and resuming of viewing when viewing a three-dimensional video virtual object.

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] (1) An information processing device worn by a user includes a processor that controls playback of a three-dimensional video virtual object, a positioning sensor that detects the user's position, a gaze sensor that detects the user's gaze direction, and a memory. The processor records user information in the memory, the user information including a placement position for placing the played-back video virtual object and at least one of the user's position detected by the positioning sensor and the user's gaze direction detected by the gaze sensor. The processor also controls playback of the video virtual object to stop when the positioning sensor detects that the user has left a range that is a predetermined distance from the placement position, and controls playback of the video virtual object to resume when the user enters the range after playback of the video virtual object has stopped, based on the user information recorded in the memory and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor.

[0010] (2) In the information processing device described in (1), when the user enters the above-mentioned range after stopping the playback of the video virtual object, if the position detected by the positioning sensor and the gaze direction detected by the gaze sensor correspond to the user information recorded in the memory, the processor resumes the playback of the video virtual object.

[0011] (3) In the information processing device described in (1), the processor controls the device to notify the user when the user enters the above-mentioned range after stopping playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory.

[0012] (4) In the information processing device described in (1), when the change in the user's gaze direction detected by the gaze sensor is below a threshold value for a predetermined period of time, the processor records the range of the video virtual object in the gaze direction in memory as a gaze range, and when the user enters the above range after stopping the playback of the video virtual object and the user's gaze direction enters the gaze range, the processor resumes playback of the video virtual object.

[0013] (5) In the information processing device described in (1), the processor records in memory the position and direction suitable for viewing the video virtual object as preferred placement information, and when the user enters the above-mentioned range after stopping playback of the video virtual object, notifies the user of the preferred placement information recorded in memory.

[0014] (6) The information processing device described in (1) includes a display that displays a video virtual object played by a processor, and when the user enters the above-mentioned range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the processor controls the video virtual object to rotate and display it on the display.

[0015] (7) The information processing device described in (1) is equipped with a display, and the processor controls the display so that a video virtual object is superimposed on a physical object, and when the user enters the above-mentioned range after stopping playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the processor controls the display so that the video virtual object is rotated according to the shape or size of the physical object and displayed on the display.

[0016] (8) In the information processing device described in (1), when the user enters the above-mentioned range after stopping playback of the video virtual object, the processor controls the video virtual object to return to a position a predetermined time before the position where playback of the video virtual object was stopped and resume playback of the video virtual object.

[0017] (9) An information processing device according to (1), wherein the processor records a plurality of playback positions in memory as positions at which the video virtual object is played, and when the user enters the above-mentioned range after stopping the playback of the video virtual object, notifies the user of the plurality of playback positions and controls the playback of the video virtual object to resume from the playback position selected by the user.

[0018] (10) In the information processing device described in (1), when the user does not enter the above-mentioned range even after a predetermined time has elapsed after stopping the playback of the video virtual object, the processor notifies the user whether or not to resume playback of the video virtual object, and when the user selects to resume playback, controls the video virtual object whose playback has been resumed to be placed in a position different from the placement position recorded in the memory.

[0019] (11) An information processing device that has the function of generating and displaying a three-dimensional video virtual object and is worn by a user, comprising a processor and a memory unit, wherein the processor records in the memory unit relative positional and directional information, which is the relative positional and directional relationship between the video virtual object and the user's position and line of sight, and when the user moves away from the video virtual object and the distance between the video virtual object and the user becomes a predetermined distance or more, stops playing the video virtual object, and after a predetermined time has elapsed since the stopping, positions the video virtual object so that it follows the movement of the information processing device in a state based on the relative positional information recorded in the memory unit, and controls the playback operation of the positioned video virtual object.

[0020] (12) An information processing device according to (11), characterized in that the processor notifies the user whether or not to play back the action of the placed video virtual object, and if the notified user chooses to play back the action of the placed video virtual object, executes the playback action of the placed video virtual object.

[0021] (13) An information processing device according to (11), characterized in that when the processor identifies that the size of the video virtual object is larger than a predetermined threshold, the processor reduces the size of the video virtual object and places it.

[0022] (14) An information processing device according to (11), characterized in that the processor records in a memory unit the gaze point at which the user was gazing at the video virtual object before stopping, and when it is determined that the size of the video virtual object is larger than a predetermined threshold, it places a video virtual object that is an excerpt of a predetermined area including the gaze point recorded in the memory unit.

[0023] (15) An information processing method for a device worn by a user, which plays a three-dimensional video virtual object, displays the played video virtual object by arranging it so as to be superimposed on a real object, records user information including a placement position of the video virtual object and at least one of the user's position and line of sight, stops playing the video virtual object when it is detected that the user has left a range that is a predetermined distance from the placement position, and resumes playing the video virtual object based on the recorded user information when the user enters the range after playing the video virtual object has been stopped.

[0024] By using the technology of the present invention, it is possible to realize an information processing device and an information processing method that allow users to easily and conveniently stop and resume viewing of a stereoscopic video virtual object. In addition, problems, configurations, and advantages other than those described above will become clear from the description of the following embodiments.

[0025] 4A is a diagram illustrating an example of a schematic external view of an example of an information processing device according to the present embodiment; FIG. 4B is a diagram illustrating an example of a schematic external view of a specific display screen of the embodiment described in FIG. 1; FIG. 4C is a diagram illustrating an example of a control operation of the information processing device according to the present embodiment shown in FIG. 1; FIG. 4D is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4A; FIG. 4E is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4C; FIG. 4F is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4F; FIG. 4G is a diagram illustrating a flowchart illustrating a basic operation of the information processing device according to the present embodiment; FIG. 4H is a diagram illustrating a recording operation of the information processing device according to the present embodiment shown in FIG. 4C; FIG. 4H is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4F; FIG. 4I is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4I; FIG. 4I is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 4I; FIG. 2 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 3 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 5 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 6 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 7 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. 1. FIG. 8 is a block diagram illustrating an example configuration of an HMD as an example of an information processing device according to the present embodiment.

[0026] An example of an embodiment of the present invention will be described below with reference to the drawings. As a specific example of an information processing device according to this embodiment, a head-mounted display (HMD) that is worn on a user's head and displays and allows visual recognition of information on real space and virtual space will be described. FIGS. 1, 3, 4(a), 5(a), 5(b), and 7 are examples of diagrams schematically illustrating the exterior of an information processing device according to this embodiment, and FIGS. 2 and 4(b) are examples of diagrams illustrating a display screen within the field of view of the information processing device according to this embodiment. In FIG. 1, an HMD 200 is worn on the head of a user 10. The HMD 200 includes a left-eye gaze sensor 201 and a right-eye gaze sensor 202 that detect the gaze of the user 10's left and right eyes, a camera 203 that captures the outside world, a positioning sensor 215 that detects the user's position on Earth, and a geomagnetic sensor 216 that detects the direction in which the HMD is facing.

[0027] In contrast to optical see-through HMDs, in which the user 10 directly sees real objects in the forward surrounding field of view, video see-through HMDs capture images of real objects in the forward surrounding field of view using a camera 203 and display the captured images of the real objects on a display. In addition to the real objects, the HMD 200 generates three-dimensional video virtual objects and displays them in a three-dimensional arrangement within the user 10's field of view. This allows the user 10 to view the video virtual objects as if the virtual objects were actually present in the real world. The HMD 200 also records information regarding the position and orientation of the placed three-dimensional video virtual objects. FIG. 1 shows a case in which a video virtual object 205, which is a video of a soccer game, is placed in the space above a desk 204, which is a real object. Although not shown, it is preferable to display a selection screen that allows the user 10 to select the real object on which to place the video virtual object, allowing the user 10 to place it in any location. The HMD 200 also detects the position of the user 10 using a positioning sensor 215 provided therein, and detects the direction in which the user 10 is facing using a geomagnetic sensor 216 provided therein.

[0028] When the user 10 is located at a position 207 inside a control area range 206 that is within a predetermined distance from the video virtual object 205 as shown in FIG. 1 , the user 10 can view the video virtual object 205. That is, as shown in FIG. 2 , the user 10 can view a video of a soccer game, which is an example of the video virtual object 205, placed on a desk 204, on a display screen 208 within the field of view. Note that the desk 204 may be a virtual object separate from the video virtual object 205. At this time, that is, when the video virtual object 205 is displayed, the HMD 200 can grasp the position and direction of the video virtual object 205 placed in three-dimensional space. Furthermore, the HMD 200 can detect the position and direction of the user 10 using a positioning sensor 215 and a geomagnetic sensor 216, and the line of sight of the user 10 using a left-eye gaze sensor 201 and a right-eye gaze sensor 202. Therefore, the HMD 200 can identify and acquire the relationship between the position and direction of the video virtual object 205 and the user 10, and the relationship between the direction in which the video virtual object 205 is displayed and the line of sight of the user 10.

[0029] Here, the user 10 moves a predetermined distance from the video virtual object 205 to a position 209 outside the control area range 206, as indicated by the arrow 210 in FIG. 3 . At this time, the HMD 200 temporarily suspends the operation of the video virtual object 205, which is a soccer game video. This allows the user 10 to immediately suspend the operation of the video virtual object 205 simply by moving outside the control area range 206, without requiring any special operation by the user. The HMD 200 also identifies the relationship between the positions and directions of the video virtual object 205 and the user 10, as well as the relationship between the direction in which the video virtual object 205 is displayed and the line of sight of the user 10, and records the relative position and direction relationship before the suspension as mutual arrangement information. Here, the HMD 200 records the information described below as the mutual arrangement information. For example, the HMD 200 records the relative position and direction relationship with respect to the video virtual object 205 at the position where the user 10 stayed before leaving the control area range 206 (the starting position of the movement when the user 10 moved outside the control area range 206). Furthermore, if the user 10 moves outside the control area range 206 without facing the direction of the video virtual object 205 for a predetermined time, the HMD 200 records the information described below as mutual placement information. For example, the HMD 200 records the position where the user 10 stayed before leaving the control area range 206 and the direction from the position toward the video virtual object 205. In this way, the HMD 200 extracts and acquires the viewing position and direction of the user 10 with respect to the video virtual object 205 before pausing, from information regarding the position of the user 10 with respect to the video virtual object 205 or the line of sight direction of the user 10 recorded before pausing. Then, the HMD 200 records the viewing position and direction of the user 10 relative to the video virtual object 205 before pausing as relative position information. Note that the stay position can be, for example, the position where the user 10 stays when viewing the video virtual object, that is, the position where the user 10 continues to view the video virtual object 205 without moving for a predetermined time. The relative position information may also be recorded as position and direction information having a predetermined range.This allows the information to be recorded as information indicating a rough position and direction. Note that instead of or in addition to recording as mutual placement information, the placement position where the video virtual object 205 is placed and the user's position and line of sight direction may be recorded as user information.

[0030] After the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 as indicated by the arrow 211 in FIG. 3 . At this time, if the user 10 approaches within a predetermined distance from the video virtual object 205 and is positioned and facing the same viewing position and direction as before the pause, the HMD 200 resumes the playback operation of the video virtual object 205 of the soccer game. In other words, if the user 10 is positioned within a predetermined range based on the recorded relative position information and faces the same direction as the recorded relative position information, the HMD 200 resumes the playback operation of the video virtual object 205. In other words, the HMD 200 determines whether the user 10 is positioned and facing the same direction as before the pause, based on the relative position information recorded in the state before the pause and information regarding the position of the user 10 and the line of sight direction of the user 10 after approaching within the predetermined distance.

[0031] This allows the user 10 to immediately resume playback of the video virtual object 205 without any special operation. Furthermore, the user 10 can resume viewing and playback of the video virtual object 205 in a state where the viewing state before the user 10 stopped the video virtual object 205 is roughly reproduced. Thus, the HMD 200 can reduce the difference in the image perceived by the user 10 before and after stopping, which occurs due to differences in the viewing direction of the user 10, for the video virtual object 205, which looks different depending on the angle. Then, the user 10 can continuously view the video virtual object 205.

[0032] On the other hand, after the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 from a position and direction different from the viewing position and direction relative to the video virtual object 205 before the pause, as indicated by arrow 221 in FIG. 4A. Then, when the user 10 reaches position 222, the HMD 200 generates and displays a message 223 on the display screen 208 within the field of view, as shown in FIG. 4B, indicating that the viewing position and direction of the user 10 are different from the viewing position and direction of the user 10 before the pause, to notify the user 10. Here, the HMD 200 can determine whether the viewing position and direction are different from the viewing position and direction before the pause based on the relative position information recorded in the state before the pause. FIG. 4B shows the display screen 208 within the field of view of the HMD 200, and illustrates an example of the message 223 displayed on the display screen 208, in which a message with the content "Notification message: Viewing position and direction are different..." is generated and displayed. This allows the HMD 200 to notify the user 10 that the position and direction of the user 10's re-entry into the control area range 206 is different from the viewing position and direction of the video virtual object 205 before pausing. Furthermore, the HMD 200 can notify the user 10 that, for the video virtual object 205, which looks different depending on the angle, the video perceived by the user before pausing and after playback will differ depending on the viewing direction. The notification method is not particularly limited, and may be a message as shown in FIG. 4B , a symbol indicating this, or a warning using sound or vibration. The HMD 200 may, for example, display a line indicating the control area range 206 as a thick or solid line, change the color of the displayed line, or flash the line for a predetermined period of time to notify the user.

[0033] Upon receiving such a notification, user 10 can recognize that he or she is in a different position from the viewing position before the pause or facing a different direction from the viewing direction before the pause. After recognizing this, user 10 can select whether or not to resume playback of the operation of video virtual object 205 as is, or whether to resume playback by controlling video virtual object 205, such as by rotating it. If user 10 selects to resume playback of the operation of video virtual object 205 as is, an effect is achieved in which user 10 can resume viewing of video virtual object 205 from the position and direction that user 10 wants to view.

[0034] When the user 10 moves a predetermined distance away from the video virtual object 205 and outside the control area range 206 as indicated by the arrow 231 in FIG. 5A and the HMD 200 stops the operation of the video virtual object 205, information about the user 10's line of sight is recorded. That is, as shown in FIG. 5A, the HMD 200 extracts a predetermined gaze range 233 centered on a gaze point 232 of the user 10 before stopping, which was identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, and records this as predetermined gaze range information. Note that if the user 10's line of sight with respect to the video virtual object 205 does not move within a predetermined time, the HMD 200 identifies a location including the position of the gaze as the gaze point 232. Alternatively, if the amount of movement of the user 10's line of sight with respect to the video virtual object 205 is equal to or less than a predetermined threshold, the HMD 200 identifies a location including the position of the gaze as the gaze point 232.

[0035] After pausing, the user 10 moves into the control area range 206 as indicated by the arrow 234 in FIG. 5B . The HMD 200 then identifies the position and orientation corresponding to the recorded relative position information, and when it identifies that the user's gaze 235 has entered the predetermined gaze range 233 indicated by the predetermined gaze range information, it resumes the playback operation of the video virtual object 205. This allows the HMD 200 to resume the playback operation of the video virtual object 205 with the user 10 directing the gaze 235 toward the vicinity of the gaze point 232 where the user 10 was gazing when the operation of the video virtual object 205 stopped. Therefore, even after playback is resumed, the user 10 can gaze at approximately the same location as before the pause from the viewing position and direction as before the pause, and can smoothly view the resumed video virtual object 205 without any discomfort from before the pause. Furthermore, the HMD 200 can resume viewing of the video virtual object 205 in a state where the state in which the user 10 was watching, paying particular attention to the video virtual object 205 before the pause, is reproduced. As shown in Figures 5(a) and 5(b), the video virtual object 205 is a soccer game, and the user 10 watches, for example, focusing on the area around the soccer ball. In this case, even when resuming playback from pause, the user 10 can continue watching without interruption, focusing on the area around the soccer ball where they are paying attention.

[0036] Furthermore, when the user 10 re-enters the control area range 206 after pausing the video virtual object 205, the HMD 200 may perform the display described below. That is, the HMD 200 may be controlled to display the relative position and direction between the video virtual object 205 recorded before the pause and the position of the re-entering user 10 or the line of sight of the user 10. Here, the display manner of the relative position and direction is not particularly limited as long as the respective relationships can be grasped. For example, the HMD 200 may numerically display the difference between the position of the virtual object 205 before the pause and the current position of the HMD 200. Here, if the difference becomes smaller than a predetermined amount, the HMD 200 may notify the user of this fact by displaying text information, outputting audio, or the like. For example, the HMD 200 may display a compass symbol indicating the direction in which the video virtual object 205 was displayed before the pause and a compass symbol indicating the current orientation of the HMD 200. Furthermore, the HMD 200 may be controlled to generate an image of a virtual object such as a circle, arrow, or dot indicating the user 10's gaze point 232 relative to the video virtual object 205 and superimpose it on the three-dimensional space. This allows the user 10 to immediately grasp the relative position and direction between the video virtual object 205 and the user 10's position and the user's line of sight before pausing, when the user 10 re-enters the control area range 206. The user 10 can also immediately grasp the user's gaze point 232 relative to the video virtual object 205. Thus, when viewing a three-dimensional video virtual object 205 placed in a specific location, the user 10 can easily and conveniently stop viewing or resume playback by performing an operation at a suitable timing according to the situation, without requiring any special operation by the user 10.

[0037] Next, the basic operation of the information processing device according to this embodiment will be described using a flowchart. FIG. 6 is an example of a flowchart illustrating the basic operation of the information processing device according to this embodiment. The basic operation of the information processing device (in this example, the HMD 200) is appropriately executed using hardware resources described below. In FIG. 6 , the video virtual object 205 is displayed on the HMD 200 worn by the user 10 (S701). Then, when the user 10 moves a predetermined distance away from the video virtual object 205 and out of the control area range 206 (S702), the HMD 200 pauses the playback of the video virtual object 205 (S703). At this time, the HMD 200 records the relative position and direction relationship between the video virtual object 205 and the position of the user 10 and the line of sight of the user 10 before stopping as relative position information. The HMD 200 also records the predetermined gaze range centered on the gaze point before stopping as predetermined gaze range information (S704). After this, while the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 and enters the control area range 206 within a predetermined distance (S705). Here, the HMD 200 identifies whether the user 10 is located at the viewing position before the stop and facing the direction before the stop (i.e., whether the user 10 is located within a predetermined range based on the recorded mutual placement information and facing the predetermined direction). Furthermore, the HMD 200 identifies whether the user 10's gaze is directed inside the predetermined gaze range (S706). When the HMD 200 identifies that the user 10 is located at the viewing position before the stop and facing the direction before the stop, and further that the user 10's gaze is directed inside the predetermined gaze range, the HMD 200 resumes playback of the video virtual object 205 (S707). Furthermore, if the display and playback of the video virtual object 205 is not finished, the HMD 200 returns to sequence S702 and repeats this flow. If the display and playback of the video virtual object 205 is finished, the HMD 200 ends this flow (S708).

[0038] On the other hand, if the user 10 is located at the viewing position before the video was stopped and is not facing the same direction as before the video was stopped, the HMD 200 notifies the user 10 that the position and direction are different (S709). If the user 10 receives the notification and decides to resume playback of the video virtual object 205 (S710), the user 10 inputs this information to the HMD 200 using an appropriate method, and the HMD 200 resumes playback of the video virtual object 205 (S707). Examples of input methods by the user 10 include button operations and gestures. At this time, the HMD 200 may resume playback of the video virtual object 205 as is, or may rotate the video virtual object 205 to a position suitable for viewing and then resume playback. If the user 10 determines not to resume playback of the video virtual object 205 (S710), and does not finish displaying and playing the video virtual object 205, the HMD 200 returns to sequence S702 and repeats this flow. When the display and playback of the video virtual object 205 is finished, the HMD 200 ends this flow (S708). The above operations provide an information processing device (in this example, the HMD 200) that allows the user to conveniently stop viewing or resume playback of a video virtual object without requiring any special operation by the user.

[0039] Next, an information processing device for notifying and instructing a user of the user's position and gaze point before pausing, and the preferred viewing position and direction of the video virtual object when stopped, will be described using Figures 7, 8, and 9. Note that in Figures 7, 8, and 9, parts shown in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b) and assigned the same reference numerals have the same operations as those already described in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b), and therefore detailed descriptions thereof will be omitted. In Figure 7, the HMD 200 can capture the position of the user 10 using a positioning sensor 215 and can capture the gaze point on the video virtual object 205 that the user 10 is gazing at using a left-eye gaze sensor 201 and a right-eye gaze sensor 202. Therefore, the HMD 200 notifies the user 10 of the position and gaze point of the user 10 captured before the pause as preferred viewing placement information, thereby enabling the user 10 to grasp the viewing position of the user 10 before the video virtual object 205 stopped.

[0040] Next, FIGS. 8 and 9 will be described. Note that the parts shown in FIG. 7 and denoted with the same reference numerals perform the same operations as those already described in FIG. 7 , and therefore, detailed description thereof will be omitted. In FIG. 8 , when the user 10 moves out of the control area range 206 as indicated by the arrow 210, the HMD 200 pauses the operation of the video virtual object 205, which is a soccer game video. At this time, the HMD 200 selects the position and direction of a suitable highlight of the video scene within the paused video virtual object 205 as suitable viewing arrangement information, and notifies and instructs the user 10. Here, the position and direction of the suitable highlight of the video scene is a position and direction that allows the view of a virtual object or the like that the user is gazing at in the video scene to be viewed up close without any obstructions. In the soccer game video shown as the video virtual object 205 in FIG. 8 , as an example, let's assume that the user is gazing at a soccer ball 801. Therefore, the HMD 200 selects a suitable viewing position / direction 802 that allows the user 10 to view the soccer ball as if it were right in front of the user's eyes without being obstructed by other virtual objects, as suitable viewing arrangement information, and notifies the user 10 of this selection. Thus, the HMD 200 can suggest to the user 10 a suitable viewing position / direction for a certain virtual object in the video virtual object 205 that the user 10 is gazing at. Furthermore, the user 10 can re-enter this suggested viewing position / direction and resume playback of the video virtual object 205 at a suitable highlight. In another example, the HMD 200 suggests to the user a suitable viewing position / direction that allows the user to view a certain soccer player in the video virtual object 205 that the user is gazing at from the front or diagonal, rather than from behind, and without being obstructed by other virtual objects. Therefore, playback can be resumed at a suitable highlight.

[0041] 9 , when the user 10 moves outside the control area range 206 as indicated by the arrow 210, the HMD 200 pauses the operation of the video virtual object 205, which is a soccer game video. At this time, the HMD 200 selects a suitable viewing position and direction for a specific virtual object in a video scene of the paused video virtual object 205 as preferred viewing arrangement information and notifies the user. In the soccer game video shown as the video virtual object 205 in FIG. 9 , as an example, a soccer player 901 is selected as the specific virtual object. The HMD 200 extracts the soccer player 901, which is surrounded by a rectangular frame 902, as the specific virtual object. The HMD 200 then selects a suitable viewing position and direction 903, which allows the user 10 to view the movement of the specific virtual object, soccer player 901, as if it were right in front of the user's eyes and as if they were holding it in their hands, as preferred viewing arrangement information, and notifies the user 10. Thus, the HMD 200 can suggest to the user 10 a position and direction in which the user 10 can preferably view a virtual object, such as a virtual object or an important person, designated in advance by the user 10 in the video virtual object 205 being viewed, regardless of the user 10's point of gaze. Furthermore, the user 10 can re-enter the suggested viewing position and direction and resume playback of the video virtual object 205 at a preferred highlight. Note that when selecting the above-mentioned specific virtual object or a viewing position and direction in which the specific virtual object can be preferably viewed, the HMD 200 may make a judgment based on the following explanation. That is, the HMD 200 may record highlight scenes frequently watched by viewers or specific virtual objects that the viewer frequently looks at for the soccer video that is the video virtual object 205, and make a judgment based on this record.

[0042] As another example, the user 10 may designate a specific virtual object while viewing the video virtual object 205, and the HMD 200 may select a suitable viewing position and direction for the designated specific virtual object as suitable viewing arrangement information and notify or instruct the user 10. In this way, the HMD 200 can notify or instruct the user 10 of a suitable viewing position and direction for the specific virtual object arbitrarily selected by the user 10.

[0043] Next, an information processing device for performing posture control, such as rotating a video virtual object, when resuming playback after the video virtual object has been stopped will be described using FIGS. 10( a), 10(b), 11, and 12. In FIGS. 10( a), 10(b), 11, and 12, parts shown in FIGS. 1-9 and labeled with the same reference numerals perform the same operations as those already described in FIGS. 1-9, and therefore detailed descriptions thereof will be omitted. In FIG. 10( a), when the user 10 moves out of the control area range 206 as indicated by arrow 210, the HMD 200 pauses the operation of the video virtual object 205, which is a soccer game video. Furthermore, the HMD 200 records, as relative position information, the relative position and direction relationship between the video virtual object 205 and the user 10 and the user 10's line of sight before the video virtual object 205 is stopped. After the pause, the user 10 approaches within a predetermined distance from the video virtual object 205 in direction 1001. Here, the HMD 200 rotates the video virtual object 205 in accordance with the user's 10 approach direction 1001 and the user's 10 position after approach 1002, based on the recorded position of the user 10 at the time of stoppage and the position and direction of the user's 10 line of sight. That is, as shown in FIG. 10( a), the user 10 approaches the soccer game video, which is the video virtual object 205, from a position 1002 on the opposite side of the direction 1001 that is approximately the same as the direction 210 from which the user 10 left. In this case, the HMD 200 rotates the soccer game video by approximately 180 degrees, as indicated by arrows 1003 and 1004. As a result, the approaching user 10 can view the soccer game video in the same position as before the stoppage, as shown in FIG. 10( b). That is, both before the stoppage and after the resumption of playback, the player on the team wearing black can be seen on the left side of the field of view, and the player on the team wearing white can be seen on the right side of the field of view, allowing for a smooth transition from the stoppage to the resumption of playback without any discomfort. User 10 approaches video virtual object 205 from a direction different from the viewing direction of user 10 before video virtual object 205 stopped, or after approaching, user 10 is positioned in a direction different from the viewing direction of user 10 before video virtual object 205 stopped.Even in such a case, even if user 10 does not move to the viewing position or direction before the stop, user 10 can resume viewing video virtual object 205 in a state where the general viewing state that was in place before video virtual object 205 stopped is reproduced.

[0044] FIG. 11 illustrates a case where, after pausing, the user 10 approaches the soccer game video, which is the video virtual object 205, from a direction 1101 that is approximately perpendicular to the direction 210 from which the user 10 moved. Here, based on the position and direction relationship between the user 10 and the user 10's line of sight at the time of the recorded pause, consider rotating the video virtual object 205 by approximately 90 degrees in the direction indicated by arrows 1102 and 1103 in accordance with the user 10's approach direction 1101. In this case, as shown in FIG. 11 , the soccer game video 1104 rotated by approximately 90 degrees will extend beyond the top of the desk 204. Therefore, if the size and shape of the placement location, such as the desk 204, prevent the rotation of the video virtual object 205 from being contained within the placement location, the HMD 200 does not rotate the video virtual object 205. This prevents the video virtual object 205 from being misaligned with the placement location due to the rotation of the video virtual object 205.

[0045] FIG. 12 illustrates a case in which, after a pause, the user 10 approaches the video virtual object 205 from direction 1001, similar to the case illustrated in FIG. 10A . Here, if a specific viewing location, such as a chair 1201, is located inside the control area range 206 that encompasses the video virtual object 205, the HMD 200 does not rotate the soccer game video, which is the video virtual object 205. In this case, since there is a high possibility that the user will resume viewing the video virtual object 205 while sitting in a chair or from a specific viewing location, the HMD 200 deliberately does not rotate the video virtual object 205, thereby enabling viewing without causing any particular discomfort. Furthermore, if there are other viewers, it is beneficial not to rotate the video virtual object 205 so as not to affect their viewing. This provides the effect of controlling the playback of the video virtual object 205 and issuing notification instructions to the user 10, taking into account the situation of the viewing location of the video virtual object 205.

[0046] Next, an information processing device that rewinds the playback position (playback time position) of a video virtual object when playing it will be described with reference to FIGS. 7 and 13 . When a user approaches within a predetermined distance of a paused video virtual object, this information processing device performs processing based on the recorded mutual placement information, information regarding the playback position of the video virtual object 205, and information regarding the user's position and the user's line of sight after approaching within the predetermined distance. Note that in FIG. 13 , parts with the same reference numerals shown in FIGS. 1 to 5 and 7 to 12 perform the same operations as those already described in those figures, and therefore detailed descriptions thereof will be omitted. When the user 10 is within the control area range 206, as shown in FIG. 7 , the user 10's line of sight is directed toward the video virtual object 205. If the user 10 subsequently attempts to move outside the control area range 206, the user 10's line of sight will move away from the video virtual object 205. Here, the HMD 200 records the playback position of the video virtual object 205 when the user 10 averts their line of sight. Furthermore, when the user 10 moves outside the control area range 206, the operation of the video virtual object 205 stops. Therefore, when the user 10 approaches the paused video virtual object 205, the HMD 200 performs the processing described below based on the recorded mutual arrangement information, information about the playback position of the video virtual object 205, and information about the user's position and the user's line of sight after approaching within a predetermined distance. In other words, when the HMD 200 resumes playback of the operation of the video virtual object 205 based on this information, it does not play from the playback position where the operation of the video virtual object 205 stopped (the position when the video stopped). The HMD 200 plays the video virtual object 205 from the playback position of the video virtual object 205 when the user 10 looks away from the video virtual object 205 or from a predetermined time before the look-away. In other words, the HMD 200 plays the video virtual object 205 from a playback position that is earlier in time than the playback position where the operation of the video virtual object 205 stopped, and that is a predetermined time before the time when the user 10 looked away or moved away from the direction of their line of sight.This allows the user 10 to view the video that was played between the time the user 10 looked away from the video virtual object 205 and the time the user 10 left the control area range 206 .

[0047] 13 is a diagram used to explain an example of processing performed by an information processing device when a user approaches within a predetermined distance of a paused video virtual object, based on recorded mutual placement information and information about the user's position and the user's gaze direction after approaching within the predetermined distance. That is, this diagram explains an example of a case in which, when playing back a video virtual object, the information processing device plays back the video virtual object from a playback position that is rewinded by the amount of time the user moved from the user's current position within the control area range to the position where the user left the control area range. In FIG. 13 , user 10 is at position 207 within control area range 206 and is watching the operation of video virtual object 205. After that, as indicated by arrow 1301, when user 10 moves to position 1302 and leaves control area range 206 without directing his or her gaze toward video virtual object 205 for a predetermined time, the operation of video virtual object 205 stops. Then, when the user 10 approaches the paused video virtual object 205, the HMD 200 plays back the video virtual object 205 based on the recorded mutual position information and information about the user's position and the user's line of sight after approaching within a predetermined distance. However, if the operation of the video virtual object 205 is played back from a playback position where the user 10 moves their line of sight away from the video virtual object 205, the resume playback position may not be the resume playback position that the user 10 considers preferable. In such a case, the resume playback position that the user 10 considers preferable may be, for example, the playback position when the user 10 starts moving from position 207 toward position 1302. Therefore, the HMD 200 records the time until the user 10 leaves the control area range 206 without directing their line of sight toward the video virtual object 205 and the time of user movement 1303 from the user's stay position to the position where they leave the control area range 206. Then, when the user 10 moves to position 1302 and leaves the control area range 206 without directing his / her gaze at the video virtual object 205 for a predetermined time, the HMD 200 plays the video virtual object 205 from the playback position rewound by the user's movement time.Therefore, when the user 10 approaches within a predetermined distance from the paused video virtual object, the HMD 200 performs uninterrupted playback based on the recorded mutual position information and information regarding the position of the user 10 after approaching within the predetermined distance and the direction of the user 10's gaze. Therefore, when the video virtual object is played back, the user 10 can view the operation of the video virtual object 205 without interruption from the playback position of the video virtual object 205 that the user 10 was viewing at position 207. The HMD 200 can determine the user's viewing importance of the video virtual object 205 based on information regarding the time it takes for the user 10 to exit the control area range 206 without directing their gaze toward the video virtual object 205. The HMD 200 may then appropriately select the playback position based on this determination. The HMD 200 can perform playback control, such as selecting the rewind time, to play back the video virtual object 205 in a form rewound by the user's movement time or to play back from the playback position when the user 10 took their gaze off the video virtual object 205.

[0048] Next, an information processing device that, when approaching within a predetermined distance from a paused video virtual object, presents multiple playback position candidates as playback positions for the video virtual object and selects and notifies the user of a playback position based on the viewing importance of the playback position of the video virtual object, will be described with reference to FIG. 14. In FIG. 14, parts shown in FIGS. 1-5 and 7-13 and assigned the same reference numerals perform the same operations as those already described in those figures, and therefore detailed descriptions thereof will be omitted. As shown in FIG. 14, when the user 10 approaches a paused video virtual object and moves into the control area range 206, the HMD 200 selects multiple playback position candidates, such as user positions 1401, 1402, and 1403, as positions from which to view the playback of the video virtual object 205. Here, the HMD 200 notifies the user by displaying a message image indicating the reason for selecting each playback position candidate and information regarding the playback position of the video virtual object. The HMD 200 generates images of virtual objects such as circles, arrows, and dots that indicate the positional and directional relationships (viewing positions and directions) between the video virtual object and the user's position and line of sight at each playback position candidate, as well as the gaze point, and superimposes the images on a three-dimensional display to provide notification. The user 10 then positions and faces one of the selected playback position candidates, or gazes at the gaze point. The HMD 200 then resumes playback of the video virtual object from the playback position of the playback position candidate that corresponds to the viewing position and direction relationship or gaze point. The playback position candidates are selected based on information such as the length and frequency of the user's gaze, highlight scenes (e.g., scenes with the highest total number of viewers, including information about other viewers' viewing), and the relationship between each viewing position and direction. For example, the starting playback position of a scene that the user 10 gazed at for the longest time is listed as a candidate, along with information about the viewing position and direction relationship. The viewing position and direction information of the playback position candidates is recorded as position and direction information having a predetermined range. By selecting playback position candidates in this way, the user can review important scenes when resuming viewing of the video virtual object, or if they run out of time to watch, they can resume viewing by omitting unimportant scenes.

[0049] Next, an information processing device in which a video virtual object is newly fixed and positioned after the user leaves the control area and the operation of the video virtual object is paused will be described with reference to FIG. 15 . In the example of FIG. 15 , the HMD 200 changes the position and orientation of the video virtual object in accordance with the movement of the HMD 200 while maintaining the relative position and orientation between the video virtual object 205 and the position of the user 10 and the line of sight of the user 10 at the time of the pause. Various coordinate systems are used to represent three-dimensional physical objects and virtual objects. The three-dimensional world coordinate system is a coordinate system considered in a world, which is a geometric model of real space and virtual space. The position and orientation of the video virtual object 205 do not change even when the HMD 200 moves. On the other hand, there is a coordinate system of the model itself that is suitable for representing each part of the three-dimensional model, and there is a local coordinate system of the three-dimensional model relative to the world coordinate system. In the local coordinate system of the HMD 200, the video virtual object 205 is fixed and positioned so that the position and direction of the video virtual object 205 change in accordance with the movement of the HMD 200 and follow the movement of the HMD 200. FIG. 15 shows an example of the local coordinate system of the HMD 200. That is, FIG. 15 shows a case where the position and direction of the video virtual object 205 are changed in accordance with the movement of the HMD 200 while maintaining the relative position and direction between the video virtual object 205 and the user 10 or the line of sight of the user 10 at the time of pause. In FIG. 15, parts with the same reference numerals shown in FIGS. 1 to 5 and 7 to 14 have the same operations as those already described in those figures, and therefore detailed description thereof will be omitted.

[0050] 15A illustrates a case where the user 10 moves away from the control area range 206 as indicated by arrow 1601 to position 1602, pausing the operation of the video virtual object 205, and a predetermined time has elapsed since then. In this case, the HMD 200 changes the position and direction of the video virtual object 205 in accordance with the movement of the HMD 200, while maintaining the relative positional and directional relationship between the video virtual object 205 and the user 10 or the user's line of sight at the time of the pause. The HMD 200 then fixes and places the video virtual object anew, as indicated by video virtual object 1603, and notifies the user 10 whether or not to resume the playback operation. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200, allowing the user 10 to select whether or not to make a new fixation at the position where the user 10 is gazing or whether or not to resume the playback operation. The HMD 200 then executes a new fixation and playback operation of the video virtual object 205 based on the selection result of the user 10 who received the notification. That is, the HMD 200 notifies the user 10 whether to fix and place the video virtual object 205 in the local coordinate system of the HMD 200 and play it, and controls the fixation and playback based on the selection result of the user 10 in response to the notification. Thus, the user 10 can resume viewing the video virtual object 205 while maintaining the relative position and direction between the video virtual object 205 and the user 10 or the line of sight of the user 10 at the time of pause. Therefore, even if the user 10 cannot return to the location where the video virtual object 205 was placed, the user 10 can appropriately resume viewing the video virtual object 205.

[0051] Furthermore, when the video virtual object 205 is fixedly placed in the local coordinate system of the HMD, if the size of the video virtual object 205 is larger than a predetermined threshold, the HMD 200 performs the processing described below. That is, as shown in FIG. 15( b), the HMD 200 displays a reduced video virtual object 1611 fixedly placed in the local coordinate system of the HMD 200 and resumes the playback operation of the video virtual object 1611. Similarly, when the video virtual object is fixedly placed in the local coordinate system of the HMD, if the size of the video virtual object is larger than a predetermined threshold, the HMD 200 performs the processing described below. As shown in FIG. 15( c), the HMD 200 displays a video virtual object 1612, which is an excerpt of a predetermined range including the gaze point of the video virtual object, fixedly placed in the local coordinate system of the HMD 200, and resumes the playback operation of the video virtual object 1612. Thus, the user 10 can resume viewing the playback operation of the video virtual object while maintaining a field of view other than the video virtual object. Thus, viewing can be conveniently stopped and resumed by an operation at a suitable timing according to the user's situation and the arrangement of the three-dimensional video virtual object.

[0052] Next, the fixed placement and playback operation of a video virtual object when a specific viewing location, such as a chair, or another viewer is present within the control area range and the user leaves the control area range will be described using FIG. 15( d). FIG. 15( d) illustrates a case where a specific viewing location, such as a chair 1621, and another viewer 11 are present at position 1624 within the control area range 206. In this case, when a predetermined time has elapsed since the user 10 moved out of the control area range 206, the HMD 200 duplicates the video virtual object while maintaining the positional and directional relationship between the video virtual object and the user's position and line of sight before pausing. The HMD 200 then newly fixes and places the duplicated video virtual object 1623 and notifies the user 10 whether or not to play back the operation of the video virtual object 1623. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200 that allows the user 10 to select whether or not to perform a new fixed placement or resume playback operation by gazing at the screen. The HMD 200 then controls the new fixed placement and playback operation of the video virtual object 1623 based on the selection result of the notified user 10. This allows a user who has left the control area to resume viewing the video virtual object at another location while maintaining the original position and orientation of the video virtual object by duplicating the video virtual object and playing it back. That is, while taking into account the situation of the original viewing location of the video virtual object, a user who has left the control area can view the video virtual object at another location before returning to the original placement location. Furthermore, when using an HMD, for example, multiple users each wearing an HMD may view a video of a virtual object (hereinafter, sometimes referred to as a video virtual object) together. In such cases, the video virtual object may be placed in a suitable space, and then the users may view it. Furthermore, if a user wants to view the video virtual object in the same way as when viewing a real object, a similar solution may be considered, even if the video virtual object is displayed in an obstructive position on the HMD screen.

[0053] Furthermore, when the user returns to the original position after resuming viewing of the video virtual object, the HMD 200 erases the newly fixedly placed video virtual object 1623. This leaves only the original video virtual object 205 in a fixed position, eliminating the confusion caused by unnecessary video virtual objects. Furthermore, if a specific viewing location such as a chair exists but no other viewers are present, the HMD 200 may perform the process described below. That is, the HMD 200 resumes playback from a playback position that shifts the playback time of the originally placed video virtual object 205 back by the amount of viewing time relative to the newly fixedly placed video virtual object 1632. This makes it possible to play the originally placed video virtual object and the newly fixedly placed video virtual object in a synchronized manner.

[0054] On the other hand, if a specific viewing location such as a chair or another viewer is present within the control area and resuming viewing at that location is prioritized, the HMD 200 may be controlled not to fix the video virtual object to the local coordinate system of the HMD or to not play the video virtual object. This prevents the user's actions until the video virtual object returns to its location, allowing the user to quickly return to the location of the video virtual object.

[0055] It should be noted that the settings of each control shown in FIGS. 15(b) to 15(d) may be changed according to the user's preference.

[0056] Next, a case will be described in which, when the user approaches another possible placement location, the video virtual object is newly fixed and placed while maintaining the relative positional and directional relationships between the video virtual object and the user's position and line of sight at the time of pause, and the playback operation is resumed. Figure 15( e) shows a case in which the user 10 moves away from the control area range 206 to position 1631, and then approaches a round platform 1633, which is another possible placement location within the control area range 1632, and moves to position 1634. Here, when the user 10 approaches the round platform 1633, which is another possible placement location, as shown in Figure 15( f), the HMD 200 performs the processing described below. The HMD 200 identifies whether the video virtual object 1635 can be fixed and placed on the round platform 1633, which is another possible placement location, while maintaining the relative positional and directional relationships between the video virtual object 205 and the user's position and line of sight at the time of pause. If the HMD 200 determines that a fixed placement is possible, it notifies the user 10 that a fixed placement is possible. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200, allowing the user 10 to select whether to perform a new fixed placement or resume playback operation by gazing at the screen. Based on the selection result of the notified user 10, the HMD 200 controls the new fixed placement and playback operation of the video virtual object 1635. Therefore, even if the user changes the placement location of the video virtual object, the user can resume viewing the video virtual object while maintaining the positional and directional relationship between the video virtual object and the user's position and line of sight before pausing. This allows the user to conveniently stop and resume viewing by performing an appropriate timing operation according to their situation. The round platform 1633 is an example of another possible placement location, and other examples may be used. Although the round platform 1633, which is another possible placement location, is shown as being within the control area range 1632, it may also be outside the control area range 1632. In addition, if there is a specific viewing location such as a chair or another viewer within the original control area, the HMD 200 may be controlled to notify the viewer that a new fixed placement of the video virtual object is possible in another location and not to perform fixed placement / playback control.This provides the effect of enabling notification instructions and fixed placement / playback control to be executed taking into account the situation of the original viewing location of the video virtual object.

[0057] An example configuration of an information processing device according to this embodiment will be described with reference to FIG. 16 . FIG. 16 is a block diagram showing an example configuration of an HMD as an example of an information processing device according to this embodiment. In FIG. 16 , components shown in FIGS. 1 , 3 to 5 , and 7 to 15 and assigned the same reference numerals perform the same operations as those already described in those figures. Therefore, some detailed descriptions of those components may be omitted. In FIG. 16 , the HMD 200 is configured appropriately using a left-eye gaze sensor 201, a right-eye gaze sensor 202, a camera 203, a positioning sensor 215, a geomagnetic sensor 216, an acceleration sensor 1504, a gyro sensor 1505, an operation input interface 1507, a display processing device 1508, a processor 1520, a memory 1530 storing a program 1531 and information data 1532, a vibration generating device 1541, an audio input device 1542, an audio output device 1543, and a communication device 1544, and each component is connected to each other via a bus 1550.

[0058] The camera 203 is a device that captures the surrounding field of view in front of the camera, and acquires the camera-captured image by converting light incident from a lens into an electrical signal using an image sensor. The camera 203 is appropriately installed so as to be able to capture the field of view in front of the user 10. In a video see-through type HMD, the camera 203 captures an actual object in real space, such as a desk 204, and the HMD 200 displays the captured image of the actual object on the display processing device 1508.

[0059] The left eye gaze sensor 201 and the right eye gaze sensor 202 are capable of detecting the movement and direction of the left eye and the right eye, respectively, and capturing the gaze of the user 10. Note that the process of detecting eye movement may utilize well-known techniques commonly used as eye tracking processing. For example, a method using corneal reflex is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is captured with an infrared camera, and the position on the cornea of ​​the reflected light from the infrared LED irradiation (corneal reflex) is used as a reference point, and the movement of the eye and the gaze are detected based on the position of the pupil relative to the position of the corneal reflex.

[0060] One example of the positioning sensor 215 is a Global Positioning System (GPS). This is a device that receives radio signals from artificial satellites to detect the current location, and can identify the current location of the user 10 wearing the HMD 200. As another example, a camera or a ranging sensor (described later) may be used as the positioning sensor. The current location can be determined by analyzing images captured by a camera or three-dimensional point cloud data measured by a ranging sensor, and techniques such as SLAM (Simultaneous Localization and Mapping) are known. Alternatively, a geomagnetic sensor, acceleration sensor, or gyro sensor (described later) may be used as the positioning sensor. The current location can be determined as a relative position from a reference position, and techniques such as PDR (Pedestrian Dead Reckoning) are known. Positioning may also be performed using Wi-Fi. The current location can be determined by detecting differences in the strength and arrival time of radio waves from multiple Wi-Fi access points and performing three-point positioning.

[0061] The geomagnetic sensor 216 is a sensor that measures the Earth's magnetic field and detects the direction of a user wearing the HMD 200, and can detect the direction in which the user is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions can be used, and by capturing changes in the geomagnetism in response to the movement of the HMD 200, it is also possible to detect the movement of the HMD 200.

[0062] The acceleration sensor 1504 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, shock, and the like. When the only acceleration applied is gravity, the acceleration sensor 1504 can calculate the tilt angle using the gravity vector and its projection on the acceleration sensor axis, and measure and detect the degree of tilt with respect to the ground. The acceleration sensor 1504 provided in the HMD 200 can detect the tilt of the HMD 200. Furthermore, the gyro sensor 1505 is a sensor that detects the angular velocity in the rotation direction, and can capture the vertical, horizontal, and diagonal orientation states. The gyro sensor 1505 can measure and detect the amount of movement in each direction. Therefore, the orientation of the HMD 200, such as its direction, can be detected using the acceleration sensor 1504 and gyro sensor 1505.

[0063] The ranging sensor 1503 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. For example, a LiDAR (Light Detection and Ranging) sensor is used, which irradiates an object with laser light such as infrared light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object. A TOF (Time of Flight) sensor may also be used, which measures the distance by measuring the reflection time of pulsed light irradiated on the object for each pixel. A millimeter-wave radar may also be used, which emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The ranging sensor 1503 measures the distance and direction to a physical object such as the desk 204, and the HMD 200 can confirm, for example, that the video virtual object 205 is placed on the desk 204 based on the measurement information.

[0064] The processor 1520 is configured using an appropriate semiconductor device such as a CPU. The processor 1520 controls each component of the HMD 200 by executing an operating system (OS) 1533 and an application program 1534 for operation control, etc., stored in the memory 1530, thereby realizing functions of the OS, middleware, applications, etc., and other functions. The HMD 200 includes a virtual object processing unit 1521, a playback control processing unit 1522, a mutual placement information processing unit 1523, a predetermined gaze range processing unit 1524, a preferred viewing placement processing unit 1525, a playback position candidate processing unit 1526, a surrounding situation processing unit 1527, and a placement feasibility processing unit 1528. These units (1521 to 1528) are considered program modules executed by the processor 1520, and the HMD 200 controls the operation of each function using these units (1521 to 1528).

[0065] The memory 1530 is configured with a nonvolatile storage device or the like, and stores various programs 1531 and information data 1532 handled by the processor 1520 or the like. The information data 1532 stores virtual object information 1535, mutual arrangement information 1536, predetermined gaze range information 1537, suitable viewing arrangement information 1538, arrangement location surrounding information 1539, and the like.

[0066] In the case of an optical see-through HMD, the display processing device 1508 can be configured to include a projection unit that projects virtual objects and notification information to the user, and a transparent half mirror that displays the projected virtual objects in front of the user's eyes. This allows the user to view both the actual objects in the field of view in front of the user and the imaged virtual objects, as if they were floating. In the case of a video see-through HMD, the display processing device 1508 can be configured to include a display such as an LCD panel that displays the actual objects in front of the user's eyes captured by the camera 203 together with the virtual objects. This allows the user 10 to view the actual objects in the field of view image in front of the user's eyes superimposed on the virtual objects.

[0067] The operation input interface 1507 is an input means such as a keyboard, key buttons, touch keys, etc., and is configured to allow the user 10 to set and input information that he or she wishes to input. The operation input interface 1507 may be provided in a position or form within the HMD 200 that allows the user 10 to easily perform input operations, or may be separated from the main body of the HMD 200 and connected via a wired or wireless connection. The HMD 200 may also display an input operation screen on the display screen of the display processing device 1508 and capture input operation information based on the position on the input operation screen to which the user's line of sight is directed. The HMD 200 may also display a pointer on the input operation screen and capture input operation information input by the user operating the pointer using the operation input interface 1507. The user 10 may also utter a voice indicating an input operation, and the HMD 200 may collect the sound using the voice input device 1542 and capture the input operation information. Furthermore, gestures corresponding to predetermined operations may be registered, and the HMD 200 may acquire input operation information by acquiring the user's operation movements using the camera 203 or the like.

[0068] The voice input device 1542 is a device that collects external sounds and the user's own voice using a microphone and converts them into voice data. The HMD 200 can take in instruction information from the user 10's voice into the HMD 200 and easily execute operations in response to the instruction information.

[0069] The audio output device 1543 outputs audio from a speaker based on the audio data, and can provide audio notification information to the user 10. For example, if the user 10 is located or facing a different position or direction from the viewing position or direction before the pause within the control area range 206, audio is emitted to notify the user that the user's viewing position or direction is different from the viewing position or direction before the pause.

[0070] The vibration generating device 1541 generates vibrations under the control of the processor 1520, and converts into vibrations notification instruction information for the user 10 transmitted by the HMD 200. The vibration generating device 1541 transmits vibrations to the head of the user wearing the HMD 200, thereby making it possible to notify the user 10 of the notification instruction information, thereby improving usability.

[0071] The communication device 1544 is a communication interface that performs wireless communication with other devices via short-range wireless communication. The communication device 1544 includes a communication processing circuit, an antenna, etc. that correspond to various predetermined communication interfaces, and transmits and receives various information, control signals, etc. It may also include a telephone communication network.

[0072] The timer 1545 measures the time until the user 10 leaves the control area range 206 without directing their gaze toward the video virtual object 205, the time of user movement 1303 from the user's stay position to the position where they leave the control area range, the time elapsed after the user 10 leaves the control area range 206, etc. The processor 1520 records the time measured by the timer 1545 in the memory 1530.

[0073] The virtual object processing unit 1521 generates a stereoscopic video virtual object 205, such as a soccer game video, and places it in three-dimensional space. It also records information about the position and orientation of the placed video virtual object 205 in the memory 1530. If the user 10 is outside the control area range 206 for a predetermined time, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 changes the position and orientation of the video virtual object in accordance with the movement of the HMD 200 while maintaining the relative position and orientation between the video virtual object and the user or the user's line of sight at the time of pause. The virtual object processing unit 1521 then newly fixes and places the video virtual object in the local coordinate system of the HMD 200. When the video virtual object 205 is fixed and placed in the local coordinate system of the HMD 200, if the size of the video virtual object is larger than a predetermined threshold, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 reduces the video virtual object 205 or extracts and arranges a predetermined range of area including the gaze point for the video virtual object recorded by the later-described predetermined gaze range processing unit 1524. Furthermore, when the user 10 re-enters the control area range 206 after pausing the video virtual object, the virtual object processing unit 1521 performs the processing described below. That is, it generates and displays a virtual object indicating the relative position and direction between the video virtual object 205 and the user 10 or the user's line of sight before the recorded video virtual object was paused, and the user's gaze point 232 for the video virtual object 205. For example, the virtual object processing unit 1521 generates images of virtual objects such as circles, arrows, and dots indicating the above information and displays them superimposed on three-dimensional space. Furthermore, it generates images of virtual objects of message images indicating information about the reason for selecting a playback position candidate for the video virtual object or the playback position of the video virtual object and displays them superimposed on three-dimensional space. Furthermore, images of virtual objects such as circles, arrows, and points that indicate information about the relationship between the video virtual object at the playback position candidate and the user's positions and directions (each viewing position and direction) are generated and superimposed on three-dimensional space.

[0074] When the user 10 moves a predetermined distance away from the video virtual object 205, the playback control processing unit 1522 pauses the operation of the video virtual object 205. Then, when the user 10 approaches within a predetermined distance from the video virtual object 205, the playback control processing unit 1522 performs the processing described below. That is, the playback operation of the paused video virtual object 205 is controlled based on the recorded mutual placement information, information regarding the user's position and the user's line of sight after approaching within the predetermined distance, information regarding a candidate playback position for the video virtual object 205, and the like. Examples of controlling the playback operation include playing the video virtual object 205 as is, or rotating the video virtual object 205 and playing it. Furthermore, the video virtual object 205 may be rewound a predetermined time and played back based on information regarding the user's viewing importance for the video virtual object and the viewing importance of the playback position of the video virtual object. The playback control processing unit 1522 also controls the playback and stop operations of the video virtual object 205 fixedly positioned in the local coordinate system of the HMD 200.

[0075] The mutual placement information processing unit 1523 acquires the relative position and direction relationship between the video virtual object and the user's position and the user's line of sight before stopping the video virtual object, generates mutual placement information, and records it in the memory 1530. For example, as the mutual placement information, the mutual placement information records the position and direction of the user 10 or the user's line of sight with respect to the video virtual object 205 at the stay position where the user 10 stayed before leaving the control area range 206. Here, as an example, the stay position is the starting position of the movement of the user 10 when the user 10 moves outside the control area range 206. This stay position may be a position where the change in sensor value is smaller than a predetermined value based on information from various sensors (e.g., the positioning sensor 1503, the acceleration sensor 1504, and the gyro sensor 1505) for a predetermined time, and it is determined that the user 10 is not moving. Furthermore, if the user 10 moves outside the control area range 206 without facing the video virtual object 205 for a predetermined time, the mutual placement information processing unit 1523 performs the processing described below. That is, the mutual placement information processing unit 1523 records the position where the user 10 stayed before leaving the control area range 206 and the direction from the position to the video virtual object 205. Here, the direction from the position to the video virtual object 205 can be acquired as appropriate. As an example, the orientation of the HMD 200 displaying the video virtual object 205 before pausing may be detected by the acceleration sensor 1504 or the gyro sensor 1505, and the detected orientation of the HMD 200 may be recorded as the direction from the position to the video virtual object 205. Note that the direction of a virtual object or the like that is superimposed may also be taken into consideration. In this way, the mutual placement information processing unit 1523 records the mutual placement information using information about the position of the user 10 relative to the video virtual object 205 and the line of sight of the user 10 that was acquired and recorded before the video virtual object 205 was paused. That is, the mutual placement information processing unit 1523 selects and acquires the viewing position and direction of the user 10 relative to the video virtual object 205 before pausing from the information, and records it as mutual placement information. Furthermore, the relative position information may be recorded as position and direction information having a predetermined range.

[0076] The predetermined gaze range processing unit 1524 selects a predetermined gaze range 233 centered on a user gaze point 232 before stopping, identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, generates predetermined gaze range information, and records it in the memory 1530. For example, when the gaze position of the user 10 relative to the video virtual object 205 does not move within a predetermined time, the predetermined gaze range processing unit 1524 identifies a location including the stationary gaze position as the gaze point 232. Alternatively, when the amount of movement of the gaze position of the user 10 relative to the video virtual object 205 within a predetermined time is equal to or less than a predetermined threshold, the predetermined gaze range processing unit 1524 identifies a location including a position where the gaze movement is equal to or less than the predetermined threshold as the gaze point 232.

[0077] The preferred viewing arrangement processing unit 1525 selects a preferred viewing position and direction of the video virtual object as preferred viewing arrangement information and records the selected preferred viewing arrangement information in the memory 1530. For example, the preferred viewing arrangement information may include the user's position and gaze point before pausing. Also, the preferred viewing position and direction may include a preferred viewing position and direction relative to a scene of a virtual object, etc., that the user gazed at in a video scene of the video virtual object at the time of pausing. Also, the preferred viewing position and direction may include a preferred viewing position and direction relative to a specific virtual object, etc., that the user gazed at in a video scene of the video virtual object at the time of pausing, or that the user specified in advance. Here, the gaze point and the gazed virtual object, etc., may be acquired, for example, based on the left eye gaze sensor 201 and the right eye gaze sensor 202. The direction relative to the specific virtual object may be estimated by tracking the specific virtual object. The user's viewing position is a position where the target gaze point, etc., can be viewed optimally, and may be estimated, for example, based on the position of each virtual object.

[0078] The playback position candidate processing unit 1526 selects candidates for the video playback position at which the video virtual object is rewound and played, and records the candidates in the memory 1530. The candidates for the video playback position may be selected based on information such as the length and frequency of the user's gazing time, the highlight scene (the scene with the most total views, including information about the viewing of other viewers), etc. The candidates for the video playback position may also be selected based on information such as the relationship between the video virtual object at the playback position candidate and the user's position or the user's line of sight (viewing position / direction). In addition to such viewing conditions, candidates may also be selected based on the content of the video virtual object (for example, information about a highlight tag indicating a highlight assigned to the video virtual object, or information about the action of a target that is focused on in the highlight).

[0079] The surrounding situation processing unit 1527 records an image within the control area range captured by the camera 203 or displayed on the display processing device 1508 in the memory 1530 as placement location surrounding information 1539. Then, based on the recorded information, the surrounding situation processing unit 1527 determines whether a specific viewing location such as a chair or other viewers exists around the video virtual object (within the control area range). Note that the objects to be determined as specific viewing locations such as chairs or other viewers may be controlled so that they can be applied not only to real objects or people but also to virtual objects other than the video virtual object.

[0080] The placement possibility processing unit 1528 analyzes the distance measurement results from the distance measurement sensor 1503 and the image captured by the camera 203 or displayed on the display processing device 1508. Then, the placement possibility processing unit 1528 identifies the distance from the user 10 to the real object or another virtual object, the size of the real object or another virtual object, the presence or absence of a horizontal surface, the type (for example, a desk or a table), etc., and determines whether the video virtual object can be placed.

[0081] With the above configuration, the HMD 200 performs the following basic operation. When the user moves a predetermined distance away from the stereoscopic video virtual object 205 arranged in three-dimensional space by the virtual object processing unit 1521, the HMD 200 causes the playback control processing unit 1522 to temporarily suspend the operation of the video virtual object. Furthermore, the mutual placement information processing unit 1523 acquires and records, as mutual placement information, the relative position and direction relationship between the video virtual object and the user or the user's line of sight before the pause. After the pause, the user approaches within a predetermined distance from the video virtual object. At this time, the playback control processing unit 1522 controls the playback operation of the video virtual object based on the recorded mutual placement information, information about the user's position and the user's line of sight after approaching within the predetermined distance, information about a candidate playback position for the video virtual object 205, and the like. The playback operation can be controlled by playing back the video virtual object as is, rotating the video virtual object, or rewinding a predetermined time before playing back.

[0082] Furthermore, in the HMD 200, the predetermined gaze range processing unit 1524 selects a predetermined gaze range centered on the user's gaze point in the state before the pause, generates and records it as predetermined gaze range information, and after the pause, if it is determined that the user's line of sight has entered the predetermined gaze range indicated by the predetermined gaze range information, the playback operation of the video virtual object is resumed.

[0083] Furthermore, in the HMD 200, when the operation of the video virtual object is stopped, the preferred viewing position / direction is selected by the preferred viewing position processing unit 1525 as preferred viewing position information, and the selected preferred viewing position information is notified to the user. The preferred viewing position information includes the user's position and gaze point before pausing, and a preferred viewing position / direction relative to a scene such as a virtual object that the user was gazing at in the video scene of the video virtual object at the time of pausing. Also included is a preferred viewing position / direction relative to a specific virtual object that the user was gazing at in the video scene of the video virtual object at the time of pausing, or a specific virtual object that was specified in advance.

[0084] Furthermore, in the HMD 200, the playback position candidate processing unit 1526 records the video playback position at which the video virtual object is rewound and played, as well as candidate viewing positions and directions at those video playback positions. The playback position candidate is selected and recorded based on gaze time information, such as the length of time the user gazes at the video virtual object and the frequency of gaze, highlight scenes (such as the scene with the most total views, including information about viewing by other viewers), and information about each viewing position and direction. That is, the playback position candidate is selected and recorded based on the viewing status of such a video virtual object and the content of the video virtual object (for example, information about highlight tags indicating highlights assigned to the video virtual object, and information about the behavior of the target that is focused on in the highlights). The playback operation of the video virtual object is resumed from the video playback position selected by the user from the selected and recorded playback position candidates.

[0085] Furthermore, the user moves a predetermined distance away from a stereoscopic video virtual object 205, such as a soccer game video, arranged in three-dimensional space by the virtual object processing unit 1521. Then, the playback control processing unit 1522 pauses the operation of the video virtual object, and the mutual placement information processing unit 1523 acquires and records the relative position and direction relationship between the video virtual object and the user's position and line of sight before the pause as mutual placement information. If the user remains a predetermined distance away from the video virtual object for a predetermined period of time, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 fixes and places the position and direction of the video virtual object in the local coordinate system of the HMD based on the mutual placement information recorded by the mutual placement information processing unit 1523 so that the position and direction of the video virtual object follow the movement of the HMD. Furthermore, the playback control processing unit 1522 controls the playback and stop operation of the video virtual object.

[0086] Furthermore, if the size of the video virtual object is larger than a predetermined threshold when the virtual object processing unit 1521 fixes and places the video virtual object in the local coordinate system of the HMD, the HMD 200 performs the following process: The video virtual object is reduced in size, or a predetermined range of area including the gaze point for the video virtual object recorded by the predetermined gaze range processing unit 1524 is extracted and fixedly placed in the local coordinate system of the HMD.

[0087] Furthermore, if the surrounding situation processing unit 1527 determines that a specific viewing location or other viewers exists around the placed video virtual object, the HMD 200 performs the following processing: That is, the virtual object processing unit 1521 does not fix and place the video virtual object in the local coordinate system of the HMD, or controls the video virtual object to be duplicated and fixedly placed in the local coordinate system of the HMD.

[0088] Furthermore, the user moves a predetermined distance away from a stereoscopic video virtual object 205, such as a soccer game video, arranged in three-dimensional space by the virtual object processing unit 1521. Then, the playback control processing unit 1522 pauses the operation of the video virtual object, and the mutual placement information processing unit 1523 acquires and records the relative position and direction relationship between the video virtual object and the user's position and line of sight before the pause as mutual placement information. If the placement possibility processing unit 1528 subsequently determines that a new video virtual object can be placed when the user approaches another possible placement location, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 controls the video virtual object to be newly fixed and placed while maintaining the relative position and direction relationship between the video virtual object and the user's position and line of sight at the time of the pause, and resumes the playback operation.

[0089] Thus, an information processing device can be realized that, when viewing a three-dimensional video virtual object placed in a specific location, can easily and conveniently stop or resume viewing by operating at an appropriate timing according to the user's situation and the placement of the three-dimensional video virtual object, without requiring any special operation by the user. For example, even if an emergency arises while a user wearing an HMD is viewing, the operation of the video virtual object can be stopped or resumed at an appropriate timing in an easy-to-use manner.

[0090] In the above explanation, an HMD has been described as a specific example of an information processing device, but the present invention applies to all devices with similar functions, and it goes without saying that the same effects and advantages can be obtained with information processing devices other than HMDs, such as smartphones and smartwatches.

[0091] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another example, or to add the configuration of another example to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each example with other configurations.

[0092] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0093] Furthermore, the above-described components, functions, processing units, processing means, etc. may be implemented in hardware, in part or in whole, by designing them as integrated circuits, for example. The above-described components, functions, etc. may also be implemented in software by a processor interpreting and executing a program that implements each function. The processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Information such as programs, tables, and files that implement each function may be stored in memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0094] The range and shape of the control area may be freely set by the user. Alternatively, the user may select an appropriate range from a plurality of pre-registered patterns. Information about the range of the control area determined by the user is stored in memory 1530 and used for processing by the HMD 200.

[0095] 10: User, 200: Head-mounted display (HMD), 201: Left eye gaze sensor, 202: Right eye gaze sensor, 203: Camera, 204: Real object (desk), 205: Video virtual object, 206: Control area range, 208: Display screen, 215: Positioning sensor, 216: Geomagnetic sensor, 1503: Distance measurement sensor, 1504: Acceleration sensor, 1505: Gyro sensor, 1507: Operation input interface, 1508: Display processing device, 1520: Processor, 1521: virtual object processing unit, 1522: playback control processing unit, 1523: mutual placement information processing unit, 1524: predetermined gaze range processing unit, 1525: preferred viewing placement processing unit, 1526: playback position candidate processing unit, 1527: surrounding situation processing unit, 1528: placement possibility processing unit, 1530: memory, 1541: vibration generating device, 1542: audio input device, 1543: audio output device, 1544: communication device, 1545: timer, 1550: bus

Claims

1. An information processing device worn by a user, comprising: a processor that controls the playback of a three-dimensional video virtual object; a positioning sensor that detects the position of the user; a gaze sensor that detects the user's gaze direction; and a memory, wherein the processor: records user information in the memory, including a placement position at which the played-back video virtual object is to be placed, and at least one of the user's position detected by the positioning sensor and the user's gaze direction detected by the gaze sensor; controls to stop the playback of the video virtual object when the positioning sensor detects that the user has left a range that is a predetermined distance from the placement position; and controls to resume the playback of the video virtual object when the user enters the range after the playback of the video virtual object has been stopped, based on the user information recorded in the memory and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor.

2. An information processing device according to claim 1, wherein the processor resumes playback of the video virtual object when the user enters the range after stopping playback of the video virtual object and if the position detected by the positioning sensor and the gaze direction detected by the gaze sensor correspond to the user information recorded in the memory.

3. An information processing device according to claim 1, characterized in that the processor controls the device to issue a notification to the user when the user enters the range after stopping playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory.

4. An information processing device according to claim 1, wherein the processor, when the amount of change in the user's gaze direction detected by the gaze sensor is below a threshold value for a predetermined period of time, records the range of the video virtual object in the gaze direction as a gaze range in the memory, and when the user enters the range after stopping playback of the video virtual object and the user's gaze direction enters the gaze range, resumes playback of the video virtual object.

5. An information processing device according to claim 1, wherein the processor records in the memory a position and direction suitable for viewing the video virtual object as suitable placement information, and when the user enters the range after stopping playback of the video virtual object, notifies the user of the suitable placement information recorded in the memory.

6. An information processing device according to claim 1, comprising a display that displays the video virtual object played back by the processor, wherein the processor controls the video virtual object to be rotated and displayed on the display when the user enters the range after playback of the video virtual object has stopped and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor differs from the user information recorded in the memory.

7. An information processing device according to claim 1, comprising a display, wherein the processor controls the video virtual object to be superimposed on a real object and displayed on the display, and when the user enters the range after stopping playback of the video virtual object and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor differs from the user information recorded in the memory, controls the video virtual object to be rotated in accordance with the shape or size of the real object and displayed on the display.

8. An information processing device according to claim 1, characterized in that, when the user enters the range after stopping playback of the video virtual object, the processor controls the video virtual object to return to a position a predetermined time before the position where playback of the video virtual object was stopped and resume playback of the video virtual object.

9. An information processing device according to claim 1, wherein the processor records a plurality of playback positions in the memory as positions at which the video virtual object is to be played, and when the user enters the range after stopping the playback of the video virtual object, notifies the user of the plurality of playback positions, and controls the playback of the video virtual object to resume from the playback position selected by the user.

10. An information processing device according to claim 1, wherein the processor notifies the user whether or not to resume playback of the video virtual object when the user does not enter the range even after a predetermined time has elapsed since playback of the video virtual object was stopped, and when the user selects to resume playback, controls the video virtual object whose playback has been resumed to be placed in a position different from the placement position recorded in the memory.

11. An information processing device that has the function of generating and displaying a three-dimensional video virtual object and is worn by a user, comprising a processor and a memory unit, wherein the processor records in the memory unit relative positional and directional information that is a relative positional and directional relationship between the video virtual object and the position and line of sight of the user, stops playback of the video virtual object when the user moves away from the video virtual object and the distance between the video virtual object and the user becomes a predetermined distance or more, and after a predetermined time has elapsed since the stopping, positions the video virtual object so that it follows the movement of the information processing device in a state based on the relative positional information recorded in the memory unit, and controls the playback operation of the positioned video virtual object.

12. An information processing device according to claim 11, wherein the processor notifies the user whether or not to replay the action of the placed video virtual object, and when the user who has received the notification selects to replay the action of the placed video virtual object, executes the replay action of the placed video virtual object.

13. An information processing device according to claim 11, wherein the processor, when identifying that the size of the video virtual object is larger than a predetermined threshold, reduces the size of the video virtual object and places it.

14. An information processing device according to claim 11, wherein the processor records in the memory unit the gaze point at which the user was gazing on the video virtual object before the stop, and when it is determined that the size of the video virtual object is larger than a predetermined threshold, it places the video virtual object in a range of a predetermined area including the gaze point recorded in the memory unit.

15. An information processing method for a device worn by a user, comprising: playing a three-dimensional video virtual object; displaying the played video virtual object by positioning it so that it is superimposed on a real object; recording user information including at least one of a position where the video virtual object is placed, the position of the user, and the direction of the user's line of sight; stopping playback of the video virtual object when it is detected that the user has left a range that is a predetermined distance from the position where the video virtual object is placed; and resuming playback of the video virtual object based on the recorded user information when the user enters the range after playback of the video virtual object has been stopped.