Information processing device and information processing method

Abstract

This information processing device which is used by being worn by a user and this information processing method involve: reproducing a stereoscopic virtual video object; disposing and displaying the reproduced virtual video object so as to be superimposed on a real object; and recording user information, which contains the disposition position where the virtual video object is disposed and / or the position and line-of-sight direction of the user. When it is detected that the user has gone out of a range which is within a prescribed distance from the disposition position, the reproduction of the virtual video object is stopped. When the user has entered said range after stoppage of the reproduction of the virtual video object, the reproduction of the virtual video object is resumed on the basis of the recorded user information.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, Augmented Reality (AR) technology has become widely used. It adds digital information to the real world, and virtual objects created using computer graphics (CG) and other technologies are reflected and augmented in the real world. Information processing devices that utilize this technology, which allow users to easily handle virtual objects while recognizing the real world in three dimensions, are becoming widespread. One example of such a device is a head-mounted display (HMD), which is worn on the head and has a display unit, camera unit, and other components.

[0003] Patent Document 1 describes the following as a method for pausing or playing the action of a video of a virtual object (hereinafter, sometimes referred to as a video virtual object). That is, Patent Document 1 states that "the playback unit 107 plays back video data stored in the storage unit 104 (301). In parallel with the playback process by the playback unit 107, the mixed reality processing unit 103 acquires the user's viewpoint position (or the user's position) and the direction of their gaze using the position and orientation estimation result by the self-position and orientation estimation unit 106 and the gaze tracking function (S302). The mixed reality processing unit 103 calculates the gaze deviation Φ from the video playback screen G and the distance D between the user's self-position and the video playback screen G. The playback unit 107 adjusts the playback speed in accordance with the deviation Φ and the distance D (S303). For example, if the deviation Φ is a predetermined angle and the distance D is equal to or greater than a predetermined value, the playback unit 107 slows down or stops the playback speed. Note that at least one of the deviation Φ and the distance D may be the condition. This is a criterion for determining whether the user is looking at the video playback screen G."

[0004] Patent Document 2 also states that "The user terminal 1500 has a function of acquiring measurement data that measures the position and attitude of the user terminal 1500, and places a subject object and a virtual camera in a virtual three-dimensional space and controls the position and attitude of the virtual camera in conjunction with the measurement data. The user terminal 1500 then generates a virtual space image taken by the virtual camera and displays it as a monitor image, while recording camerawork data 550 that can reproduce the position and attitude of the virtual camera." and "The playback image generation control unit 232 controls the generation of a playback image, which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork, based on the camerawork data 550 recorded by the recording control unit 224." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. WO2021 / 132168 A1 [Patent Document 2] Japanese Patent Publication No. 2020-119334 Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, Patent Document 1 discloses that the stopping and playback of a video virtual object screen are controlled based on the position and direction of the user's line of sight and the distance from a placed planar video virtual object. However, no consideration is given to the case of a three-dimensional video virtual object that can be viewed from multiple positions and directions. Therefore, there is a problem in stopping and resuming viewing of a three-dimensional video virtual object in an appropriate state and at an appropriate timing.

[0007] As described above, Patent Document 2 discloses that playback of a video virtual object is controlled based on information related to camerawork (camera position and orientation). However, it does not provide any information regarding the user's line of sight with respect to the video virtual object or the distance from the user. Therefore, there is a problem in that it is not possible to control the stopping and playback of a video virtual object based on the position, directional deviation, and distance of the user's line of sight with respect to the video virtual object. Furthermore, while Patent Document 2 discloses a suitable "video creation" technology, it does not disclose any description related to suitable "viewing," leaving a problem in terms of stopping and resuming "viewing" at a suitable timing. Therefore, in consideration of the above problems, the present invention aims to provide an information processing device and an information processing method that allow users to easily and conveniently stop and resume viewing of a stereoscopic video virtual object. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] (1) An information processing device worn by a user includes a processor that controls the playback of a three-dimensional video virtual object, a positioning sensor that detects the user's position, a gaze sensor that detects the user's gaze direction, and a memory. The processor records user information in the memory, the user information including a placement position for placing the played-back video virtual object and at least one of the user's position detected by the positioning sensor and the user's gaze direction detected by the gaze sensor. The processor also controls the playback of the video virtual object to stop when the positioning sensor detects that the user has left a range that is a predetermined distance from the placement position, and controls the playback of the video virtual object to resume when the user enters the range after the playback of the video virtual object has stopped, based on the user information recorded in the memory and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor.

[0010] (2) In the information processing device described in (1), when the user enters the above-mentioned range after stopping the playback of the video virtual object, the processor resumes the playback of the video virtual object if the position detected by the positioning sensor and the gaze direction detected by the gaze sensor correspond to the user information recorded in the memory.

[0011] (3) In the information processing device described in (1), the processor controls the device to notify the user when the user enters the above-mentioned range after stopping playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory.

[0012] (4) In the information processing device described in (1), when the amount of change in the user's gaze direction detected by the gaze sensor is below a threshold value within a predetermined period of time, the processor records the range of the video virtual object in the gaze direction in memory as a gaze range, and when the user enters the above range after stopping the playback of the video virtual object and the user's gaze direction enters the gaze range, the processor resumes playback of the video virtual object.

[0013] (5) In the information processing device described in (1), the processor records in memory the position and direction suitable for viewing the video virtual object as preferred placement information, and when the user enters the above range after stopping playback of the video virtual object, notifies the user of the preferred placement information recorded in memory.

[0014] (6) The information processing device described in (1) includes a display that displays a video virtual object played by a processor, and when the user enters the above-mentioned range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the processor controls the video virtual object to rotate and display it on the display.

[0015] (7) The information processing device described in (1) is equipped with a display, and the processor controls the video virtual object to be superimposed on the physical object and displayed on the display, and when the user enters the above-mentioned range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the processor controls the video virtual object to be rotated according to the shape or size of the physical object and displayed on the display.

[0016] (8) In the information processing device described in (1), when the user enters the above-mentioned range after stopping playback of the video virtual object, the processor controls the video virtual object to return to a position a predetermined time before the position where playback of the video virtual object was stopped and resume playback of the video virtual object.

[0017] (9) An information processing device according to (1), wherein the processor records a plurality of playback positions in memory as positions at which the video virtual object is played, and when the user enters the above-mentioned range after stopping the playback of the video virtual object, notifies the user of the plurality of playback positions and controls the playback of the video virtual object to resume from the playback position selected by the user.

[0018] (10) In the information processing device described in (1), when the user does not enter the above range even after a predetermined time has elapsed after stopping the playback of the video virtual object, the processor notifies the user whether or not to resume playback of the video virtual object, and when the user selects to resume playback, controls the video virtual object whose playback has been resumed to be placed in a position different from the placement position recorded in the memory.

[0019] (11) An information processing device that has the function of generating and displaying a three-dimensional video virtual object and is worn by a user, comprising a processor and a memory unit, wherein the processor records in the memory unit relative positional and directional information, which is the relative positional and directional relationship between the video virtual object and the user's position and line of sight, and when the user moves away from the video virtual object and the distance between the video virtual object and the user becomes a predetermined distance or more, stops playing the video virtual object, and after a predetermined time has elapsed since the stopping, positions the video virtual object so that it follows the movement of the information processing device in a state based on the relative positional information recorded in the memory unit, and controls the playback operation of the positioned video virtual object.

[0020] (12) An information processing device according to (11), characterized in that the processor notifies the user whether or not to play back the action of the placed video virtual object, and if the notified user chooses to play back the action of the placed video virtual object, executes the playback action of the placed video virtual object.

[0021] (13) An information processing device according to (11), characterized in that when the processor identifies that the size of the video virtual object is larger than a predetermined threshold, the processor reduces the size of the video virtual object and places it.

[0022] (14) An information processing device as described in (11), characterized in that the processor records in a memory unit the gaze point at which the user was gazing at the video virtual object before stopping, and when it is determined that the size of the video virtual object is larger than a predetermined threshold, it places a video virtual object that is an excerpt of a predetermined area including the gaze point recorded in the memory unit.

[0023] (15) An information processing method for a device worn by a user, which plays a three-dimensional video virtual object, arranges and displays the played video virtual object so as to be superimposed on a real object, records user information including a placement position of the video virtual object and at least one of the user's position and line of sight, stops playing the video virtual object when it is detected that the user has left a range that is a predetermined distance from the placement position, and resumes playing the video virtual object based on the recorded user information when the user enters the range after stopping playing the video virtual object. [Effects of the Invention]

[0024] By using the technology of the present invention, it is possible to realize an information processing device and an information processing method that allow for easy and convenient pausing and resuming of viewing of a stereoscopic video virtual object. Furthermore, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram illustrating an example of a schematic external view of an example of an information processing apparatus according to an embodiment of the present invention; [Figure 2] 2 is a diagram illustrating an example of a schematic view of the appearance of a specific display screen according to the embodiment described in FIG. 1. [Figure 3] 2 is a diagram illustrating an example of a control operation of the information processing device according to the present embodiment shown in FIG. 1. [Figure 4(a)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 4(b)] 5A and 5B are diagrams illustrating examples of the appearance of a display screen during the control operation of the information processing device according to the present embodiment shown in FIG. 4A. [Figure 5(a)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 5(b)]1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 6] 1 is an example of a flowchart illustrating a basic operation of the information processing device according to the present embodiment. [Figure 7] 2 is a diagram illustrating an example of a recording operation of the information processing device according to the embodiment shown in FIG. 1. [Figure 8] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 9] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 10(a)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 10(b)] 11A and 11B are diagrams illustrating examples of control operation results of the information processing device according to the present embodiment shown in FIG. 10A. [Figure 11] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 12] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 13] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 14] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(a)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(b)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(c)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(d)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(e)]1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 15(f)] 1. FIG. 4 is another example of a diagram illustrating the control operation of the information processing device according to the present embodiment shown in FIG. [Figure 16] 1 is a block diagram showing an example of the configuration of an HMD as an example of an information processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. As a specific example of an information processing device according to this embodiment, a head-mounted display (HMD) that is worn on a user's head and displays and allows visual recognition of information on real space and virtual space will be described. FIGS. 1, 3, 4(a), 5(a), 5(b), and 7 are examples of diagrams schematically illustrating the exterior of an information processing device according to this embodiment, and FIGS. 2 and 4(b) are examples of diagrams illustrating a display screen within the field of view of the information processing device according to this embodiment. In FIG. 1, an HMD 200 is worn on the head of a user 10. This HMD 200 includes a left-eye gaze sensor 201 and a right-eye gaze sensor 202 that detect the gaze of the left and right eyes of the user 10, a camera 203 that captures the outside world, a positioning sensor 215 that detects a position on the Earth, and a geomagnetic sensor 216 that detects the direction in which the HMD is facing.

[0027] In contrast to an optical see-through HMD in which the user 10 directly sees real objects in the forward surrounding field of view, a video see-through HMD uses a camera 203 to capture images of real objects in the forward surrounding field of view and displays the captured images of the real objects on a display. In addition to the real objects, the HMD 200 generates stereoscopic video virtual objects and displays them in a three-dimensional arrangement within the field of view of the user 10. This allows the user 10 to view the video virtual objects as if they were actually present in the real world. The HMD 200 also records information about the position and direction of the placed three-dimensional video virtual object. Fig. 1 shows a case where a video virtual object 205, which is a video of a soccer game, is placed in the space above a desk 204, which is an actual object. Although not shown, it is preferable to display a selection screen that allows the user 10 to select the actual object on which to place the video virtual object, so that the user 10 can place it in any location. The HMD 200 also detects the position of the user 10 using a positioning sensor 215 provided therein, and detects the direction in which the user 10 is facing using a geomagnetic sensor 216 provided therein.

[0028] When user 10 is at position 207 inside control area range 206, which is within a predetermined distance from video virtual object 205 as shown in FIG. 1, user 10 can view video virtual object 205. That is, as shown in FIG. 2, user 10 can view a video of a soccer game, which is an example of video virtual object 205, placed on desk 204, on display screen 208 within the field of view. Note that desk 204 may be a virtual object separate from video virtual object 205. At this time, that is, when displaying video virtual object 205, HMD 200 can grasp the position and orientation of video virtual object 205 placed in three-dimensional space. Furthermore, HMD 200 can detect the position and orientation of user 10 using positioning sensor 215 and geomagnetic sensor 216, and the gaze of user 10 using left eye gaze sensor 201 and right eye gaze sensor 202. Therefore, the HMD 200 can identify and acquire the relationship between the position and direction of the video virtual object 205 and the user 10, and the relationship between the direction in which the video virtual object 205 is displayed and the line of sight of the user 10.

[0029] Here, as indicated by arrow 210 in FIG. 3, user 10 moves a predetermined distance away from video virtual object 205 to position 209 outside control area range 206. At this time, HMD 200 temporarily suspends the movement of video virtual object 205, which is a soccer game video. This allows user 10 to immediately suspend the movement of video virtual object 205 simply by moving outside control area range 206, without requiring any special operation by the user. Furthermore, HMD 200 identifies the relationship between the positions and directions of video virtual object 205 and user 10, and the relationship between the direction in which video virtual object 205 is displayed and the line of sight of user 10, and records the relative position and direction relationship before the suspension as relative position information. Here, the HMD 200 records the information described below as the mutual arrangement information. For example, the HMD 200 records the relative position and direction relationship with respect to the video virtual object 205 at the position where the user 10 stayed before leaving the control area range 206 (the starting position of the movement of the user 10 when moving outside the control area range 206). Furthermore, if the user 10 moves outside the control area range 206 without facing the direction of the video virtual object 205 for a predetermined time, the HMD 200 records the information described below as the mutual arrangement information. For example, the HMD 200 records the position where the user 10 stayed before leaving the control area range 206 and the direction from the position to the video virtual object 205. In this way, the HMD 200 extracts and acquires the viewing position and direction of the user 10 relative to the video virtual object 205 before pausing from information about the position of the user 10 relative to the video virtual object 205 or the line of sight direction of the user 10 recorded before pausing. The HMD 200 then records the viewing position and direction of the user 10 relative to the video virtual object 205 before pausing as relative position information. As an example, the stay position can be the position where the user 10 stays when viewing the video virtual object, that is, the position where the user 10 continues to view the video virtual object 205 without moving for a predetermined period of time. The relative position information may also be recorded as position and direction information having a predetermined range. This allows it to be recorded as information indicating a rough position or direction. Instead of or in addition to recording as mutual placement information, the placement position where the video virtual object 205 is placed and the user's position and line of sight direction may be recorded as user information.

[0030] After the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 as indicated by the arrow 211 in FIG. 3 . At this time, if the user 10 approaches within a predetermined distance from the video virtual object 205 and is positioned and facing the same viewing position and direction as before the pause, the HMD 200 resumes the playback operation of the video virtual object 205 of the soccer game. That is, if the user 10 is positioned within a predetermined range based on the recorded relative position information and faces the same direction as the recorded relative position information, the HMD 200 resumes the playback operation of the video virtual object 205. That is, the HMD 200 determines whether the user 10 is positioned and facing the same direction as before the pause, based on the relative position information recorded before the pause and information regarding the position of the user 10 and the line of sight direction of the user 10 after approaching within the predetermined distance.

[0031] This allows the playback operation of the video virtual object 205 to be immediately resumed without requiring any special operation by the user 10. Furthermore, the user 10 can resume viewing and playback of the video virtual object 205 in a state where the general viewing state before the user 10 stopped the video virtual object 205 is reproduced. Thus, the HMD 200 can reduce the difference in the video recognized by the user 10 before and after stopping, which occurs due to differences in the viewing direction of the user 10, for the video virtual object 205, which looks different depending on the angle. Then, the user 10 can view the video virtual object 205 continuously.

[0032] On the other hand, after the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 from a position and direction different from the viewing position and direction relative to the video virtual object 205 before the pause, as shown by arrow 221 in Fig. 4(a). Then, when the user 10 reaches position 222, the HMD 200 creates and displays a message 223 on the display screen 208 within the field of view, as shown in Fig. 4(b), indicating that the viewing position and direction of the user 10 are different from the viewing position and direction of the user 10 before the pause, to notify the user 10. Here, the HMD 200 can determine whether the viewing position and direction are different from the viewing position and direction before the pause, based on the relative position information recorded in the state before the pause. 4(b) shows the display screen 208 within the field of view of the HMD 200, and illustrates an example of a message 223 to be displayed on the display screen 208, in which a message with the content "Notification message: Viewing position and direction are different..." is generated and displayed. This allows the HMD 200 to notify the user 10 that the position and direction of the user 10's re-entry into the control area range 206 is different from the viewing position and direction of the video virtual object 205 before pausing. Furthermore, the HMD 200 can notify the user 10 that, for the video virtual object 205, which looks different depending on the angle, the video perceived by the user before pausing and after playback will differ depending on the viewing direction. The form of the notification is not particularly limited, and may be a message as shown in Fig. 4(b), a symbol indicating the notification, a sound or vibration warning, etc. The HMD 200 may, for example, display a line indicating the control area range 206 as a thick or solid line, change the color of the displayed line, or flash the line for a predetermined period of time to notify the user.

[0033] Upon receiving such a notification, user 10 can recognize that he or she is facing a different position or direction from the viewing position before the pause. After recognizing this, user 10 can select whether or not to resume playback of the operation of video virtual object 205, or whether to resume playback by performing control such as rotation on video virtual object 205. If user 10 selects to resume playback of the operation of video virtual object 205, an effect is achieved in which user 10 can resume viewing of video virtual object 205 from the position or direction that user 10 desires to view.

[0034] When the user 10 moves a predetermined distance away from the video virtual object 205 and out of the control area range 206 as indicated by the arrow 231 in FIG. 5(a), and the HMD 200 stops the operation of the video virtual object 205, information about the user 10's line of sight is recorded. That is, as shown in FIG. 5(a), the HMD 200 extracts a predetermined gaze range 233 centered on a gaze point 232 of the user 10 before the stop, which was identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, and records this as predetermined gaze range information. Note that if the user 10's line of sight with respect to the video virtual object 205 does not move within a predetermined time, the HMD 200 identifies a location including the position of the gaze point as the gaze point 232. Alternatively, if the amount of movement of the user 10's line of sight with respect to the video virtual object 205 is equal to or less than a predetermined threshold, the HMD 200 identifies a location including the position of the gaze point as the gaze point 232.

[0035] After pausing, the user 10 moves into the control area range 206 as indicated by the arrow 234 in FIG. 5(b). The HMD 200 then identifies the position and orientation corresponding to the recorded relative position information, and when it identifies that the gaze point 235 of the user 10 has entered the predetermined gaze range 233 indicated by the predetermined gaze range information, it resumes the playback operation of the video virtual object 205. This allows the HMD 200 to resume the playback operation of the video virtual object 205 with the user 10 directing the gaze point 235 toward the vicinity of the gaze point 232 where the user 10 was gazing when the operation of the video virtual object 205 stopped. Therefore, even after playback is resumed, the user 10 can gaze at approximately the same location as before the pause from the viewing position and direction before the pause, and can smoothly view the resumed video virtual object 205 without any discomfort from before the pause. Furthermore, the HMD 200 can resume playback of the viewing of the video virtual object 205 in a state that recreates the state in which the user 10 was watching the video virtual object 205 with particular attention paid to it before the video was stopped. 5(a) and 5(b), the video virtual object 205 is a soccer game, and for example, the user 10 watches while focusing on the area around the soccer ball. In this case, even when playback is resumed from a pause, the user 10 can continue watching without interruption, focusing on the area around the soccer ball that they are focusing on.

[0036] Furthermore, when the user 10 re-enters the control area range 206 after pausing the video virtual object 205, the HMD 200 may perform the display described below. That is, the HMD 200 may be controlled to display the relative position and direction between the video virtual object 205 recorded before the pause and the position of the re-entering user 10 or the line of sight of the user 10. Here, the display manner of the relative position and direction is not particularly limited as long as the respective relationships can be grasped. For example, the HMD 200 may numerically display the difference between the position of the virtual object 205 before the pause and the current position of the HMD 200. Here, if the difference becomes smaller than a predetermined amount, the HMD 200 may notify the user of this fact by displaying text information, outputting audio, or the like. The HMD 200 may, for example, display a compass indicating the direction in which the video virtual object 205 was displayed before the pause and a compass indicating the current orientation of the HMD 200. Furthermore, the HMD 200 may generate an image of a virtual object such as a circle, arrow, or point indicating the gaze point 232 of the user 10 on the video virtual object 205, and perform control so as to superimpose and display it on the three-dimensional space. As a result, when user 10 re-enters control area range 206, user 10 can immediately grasp the relative position and direction between video virtual object 205 and user 10's position and line of sight before pausing. User 10 can also immediately grasp user gaze point 232 with respect to video virtual object 205. Thus, when viewing a three-dimensional video virtual object 205 placed in a specific location, no special operation is required by user 10, and the effect is obtained that viewing can be stopped and playback resumed simply and conveniently by an operation at a suitable timing according to the situation.

[0037] Next, the basic operation of the information processing device according to this embodiment will be described using a flowchart. FIG. 6 is an example of a flowchart illustrating the basic operation of the information processing device according to this embodiment. The basic operation of the information processing device (in this example, the HMD 200) is appropriately executed using hardware resources described below. In FIG. 6, the video virtual object 205 is played and displayed on the HMD 200 worn by the user 10 (S701). Then, when the user 10 moves a predetermined distance away from the video virtual object 205 and out of the control area range 206 (S702), the HMD 200 pauses the playback of the video virtual object 205 (S703). At this time, the HMD 200 records the relative position and direction relationship between the video virtual object 205 and the position of the user 10 and the line of sight of the user 10 before stopping as relative position information. The HMD 200 also records the predetermined gaze range centered on the gaze point before stopping as predetermined gaze range information (S704). After this, while the operation of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 and enters the control area range 206 within a predetermined distance (S705). Here, the HMD 200 identifies whether the user 10 is located at the viewing position before the stoppage and whether the user 10 is facing the direction before the stoppage (i.e., whether the user 10 is located within a predetermined range based on the recorded mutual placement information and whether the user 10 is facing the predetermined direction). Furthermore, the HMD 200 identifies whether the user 10's gaze is directed inside a predetermined gaze range (S706). If the HMD 200 identifies that the user 10 is located at the viewing position before the stoppage and facing the direction before the stoppage, and further identifies that the user 10's gaze is directed inside the predetermined gaze range, the HMD 200 resumes playback of the video virtual object 205 (S707). Furthermore, if the display and playback of the video virtual object 205 is not completed, the HMD 200 returns to sequence S702 and repeats this flow. If the HMD 200 completes display and playback of the video virtual object 205, it ends this flow (S708).

[0038] On the other hand, the user 10 is positioned at the viewing position before the stop. Do not placeIf the user 10 is not facing the same direction as before the stoppage, the HMD 200 notifies the user 10 that the position and direction are different (S709). Here, if the user 10 who has received the notification decides to resume playing the video virtual object 205 (S710), the user 10 inputs this to the HMD 200 by an appropriate method, and the HMD 200 resumes playing the video virtual object 205 (S707). Note that the input method by the user 10 may be, for example, a button operation or a gesture. At this time, the HMD 200 may resume playing the video virtual object 205 as is, or may rotate the video virtual object 205 to a position suitable for viewing and resume playing it. If the user 10 determines not to resume playback of the video virtual object 205 (S710), and does not end the display and playback of the video virtual object 205, the HMD 200 returns to sequence S702 and repeats this flow. When the display and playback of the video virtual object 205 ends, the HMD 200 ends this flow (S708). The above operations provide an information processing device (in this example, the HMD 200) that allows the user to conveniently stop viewing or resume playback of a video virtual object without requiring any special operation by the user.

[0039] Next, an information processing device for notifying and instructing a user of the user's position and gaze point before pausing and the preferred viewing position and direction of a video virtual object when pausing will be described with reference to Figures 7, 8, and 9. Note that in Figures 7, 8, and 9, parts shown in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b) and assigned the same reference numerals have the same operations as those already described in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b), and therefore some detailed descriptions thereof will be omitted. In Figure 7, the HMD 200 can capture the position of the user 10 using a positioning sensor 215 and can capture the gaze point on a video virtual object 205 that the user 10 is gazing at using a left-eye gaze sensor 201 and a right-eye gaze sensor 202. Therefore, the HMD 200 notifies the user 10 of the position and gaze point of the user 10 captured before the pause as preferred viewing placement information, thereby enabling the user 10 to grasp the viewing position of the user 10 before the video virtual object 205 stopped.

[0040] Next, we will explain Figures 8 and 9. Note that parts shown in Figure 7 and assigned the same reference numerals have the same operations as those already explained in Figure 7, so some detailed explanations of them will be omitted. In Figure 8, when user 10 moves out of control area range 206 as indicated by arrow 210, HMD 200 pauses the operation of video virtual object 205, which is a soccer game video. At this time, HMD 200 selects the position and direction of a suitable highlight of the video scene within video virtual object 205 in the paused state as suitable viewing arrangement information and notifies user 10. Here, the position and direction of a suitable highlight of the video scene is a position and direction that allows a view of a virtual object or the like that the user is gazing at in the video scene up close without any obstructions. In the soccer game video shown as video virtual object 205 in Figure 8, as an example, let's assume that the user is gazing at soccer ball 801. Therefore, the HMD 200 selects the suitable viewing position and direction 802 that allows the user 10 to view the soccer ball as if it were right in front of their eyes and as if they could hold it in their hands without it being obstructed by other virtual objects, as the suitable viewing arrangement information, and notifies the user 10 of this. Thus, the HMD 200 can suggest to the user 10 a suitable viewing position and direction for a certain virtual object in the video virtual object 205 that the user 10 has been gazing at. Furthermore, the user 10 can re-enter the suggested viewing position and direction and resume playback of the video virtual object 205 at a suitable highlight. In another example, the HMD 200 suggests to the user a suitable viewing position and direction that allows the user to view a certain soccer player in the video virtual object 205 that the user has been gazing at from the front or diagonal rather than from behind, and that allows viewing without being obstructed by other virtual objects. Therefore, playback can be resumed at a suitable highlight.

[0041] 9 , when the user 10 moves out of the control area range 206 as indicated by the arrow 210, the HMD 200 pauses the movement of the video virtual object 205, which is a soccer game video. At this time, the HMD 200 selects a suitable viewing position and direction for a specific virtual object in a video scene in which the paused video virtual object 205 is present as viewing-preferred arrangement information, and notifies and instructs the user. In the soccer game video shown as the video virtual object 205 in FIG. 9 , for example, a soccer player 901 is selected as the specific virtual object. The HMD 200 extracts the soccer player 901, surrounded by a rectangular frame 902, as the specific virtual object. The HMD 200 then selects a suitable viewing position and direction 903, which allows the user 10 to view the movement of the specific virtual object, soccer player 901, as if it were right in front of their eyes, without being obstructed by other virtual objects, as viewing-preferred arrangement information, and notifies and instructs the user 10. Thus, the HMD 200 can suggest to the user 10 a position and direction where the user 10 can preferably view a virtual object, such as a virtual object or an important person, designated in advance by the user 10 in the video virtual object 205 viewed by the user 10, regardless of the user 10's point of gaze. Furthermore, the user 10 can re-enter the suggested viewing position and direction and resume playback of the video virtual object 205 at a preferred highlight. Note that when selecting the above-mentioned specific virtual object or a viewing position and direction where the specific virtual object can be preferably viewed, the HMD 200 may make a judgment based on the following explanation. That is, the HMD 200 may record highlight scenes frequently watched by viewers or specific virtual objects that the viewer frequently looks at for the soccer video that is the video virtual object 205, and make a judgment based on this record.

[0042] As another example, the user 10 may specify a specific virtual object while viewing the video virtual object 205, and the HMD 200 may select a suitable viewing position and direction for the specified specific virtual object as suitable viewing arrangement information and notify or instruct the user 10. In this way, the HMD 200 can notify or instruct the user 10 of a suitable viewing position and direction for the specific virtual object arbitrarily selected by the user 10.

[0043] Next, an information processing device that performs posture control, such as rotating a video virtual object, when resuming playback after the video virtual object has been stopped will be described with reference to FIGS. 10(a), 10(b), 11, and 12. In FIGS. 10(a), 10(b), 11, and 12, parts denoted with the same reference numerals as those shown in FIGS. 1-9 have the same operations as those already described in FIGS. 1-9, and therefore detailed descriptions thereof will be omitted. In FIG. 10(a), when user 10 moves out of control area 206 as indicated by arrow 210, HMD 200 pauses the operation of video virtual object 205, which is a soccer game video. Furthermore, HMD 200 records, as relative position information, the relative position and direction relationship between video virtual object 205 and user 10 and the user's line of sight before pausing video virtual object 205. After the pause, user 10 approaches video virtual object 205 from direction 1001 to within a predetermined distance. Here, the HMD 200 rotates the video virtual object 205 according to the approach direction 1001 of the user 10 and the position 1002 after approaching, based on the recorded position of the user 10 when stopped and the position and direction relationship of the user's 10 line of sight. That is, as shown in FIG. 10(a), the user 10 approaches the soccer game video, which is the video virtual object 205, from a position 1002 on the opposite side in a direction 1001 that is approximately the same as the direction 210 from which the user 10 left. In this case, the HMD 200 rotates the soccer game video by approximately 180 degrees as indicated by arrows 1003 and 1004. As a result, the approaching user 10 can view the soccer game video in the same position as before it was stopped, as shown in FIG. 10(b). That is, both before it was stopped and after playback resumes, the team players in black clothing can be seen on the left side of the field of view, and the team players in white clothing can be seen on the right side of the field of view, allowing for a smooth and natural transition from stopped to restarted playback. The user 10 approaches the video virtual object 205 from a direction different from the viewing direction of the user 10 before the video virtual object 205 was stopped, or after approaching, the user 10 is positioned in a direction different from the viewing direction of the user 10 before the video virtual object 205 was stopped. Even in such cases, the user 10 can resume viewing the video virtual object 205 in a state where the general viewing state they had before the video virtual object 205 was stopped is reproduced, even if the user 10 does not move to the viewing position and direction before the video virtual object 205 was stopped.

[0044] FIG. 11 shows a case where, after pausing, user 10 approaches soccer game video, which is video virtual object 205, from a direction 1101 that is approximately perpendicular to the direction 210 from which user 10 moved. Here, based on the position and direction relationship of user 10 and user 10's line of sight at the time of the recorded pause, consider rotating video virtual object 205 by approximately 90 degrees in the direction indicated by arrows 1102 and 1103 according to user 10's approach direction 1101. In this case, as shown in FIG. 11, soccer game video 1104 rotated by approximately 90 degrees will extend beyond desk 204. Therefore, if rotating video virtual object 205 does not fit within the size or shape of the placement location, such as desk 204, HMD 200 does not rotate video virtual object 205. This prevents misalignment between video virtual object 205 and its placement location due to the rotation of video virtual object 205.

[0045] FIG. 12 illustrates a case where, after a pause, the user 10 approaches the video virtual object 205 from the direction 1001, as in the case illustrated in FIG. 10(a). Here, if a specific viewing location, such as a chair 1201, is located inside the control area range 206 that contains the video virtual object 205, the HMD 200 does not rotate the soccer game video, which is the video virtual object 205. In this case, since there is a high possibility that the user will resume viewing the video virtual object 205 while sitting in a chair or from a specific viewing location, the HMD 200 deliberately does not rotate the video virtual object 205, thereby enabling viewing without causing any particular discomfort. Furthermore, if there are other viewers, it is beneficial not to rotate the video virtual object 205 so as not to affect their viewing. This provides the effect of controlling the playback of the video virtual object 205 and issuing notification instructions to the user 10, taking into account the situation of the viewing location of the video virtual object 205.

[0046] Next, an information processing device in which the playback position (playback time position) of a video virtual object is rewound and played back when playing back the video virtual object will be described with reference to FIGS. 7 and 13. When a user approaches within a predetermined distance from a paused video virtual object, this information processing device performs processing based on the recorded mutual arrangement information, information related to the playback position of video virtual object 205, and information related to the user's position and the user's line of sight after approaching within the predetermined distance. Note that in FIG. 13, parts shown in FIGS. 1 to 5 and 7 to 12 and assigned the same reference numerals have the same operations as those already described in those figures, and therefore detailed descriptions thereof will be omitted. When user 10 is within control area range 206, as shown in FIG. 7, the gaze of user 10 is directed toward video virtual object 205. If user 10 subsequently attempts to move outside control area range 206, the gaze of user 10 moves away from video virtual object 205. Here, HMD 200 records the playback position of video virtual object 205 when user 10 moves away from the gaze. If user 10 further moves outside control area range 206, the operation of video virtual object 205 stops. Therefore, when user 10 approaches the paused video virtual object 205, HMD 200 performs the processing described below based on the recorded mutual arrangement information, information regarding the playback position of video virtual object 205, and information regarding the user's position and the user's gaze direction after approaching within a predetermined distance. In other words, when HMD 200 resumes playback of the operation of video virtual object 205 based on this information, it does not play back from the playback position where the operation of video virtual object 205 stopped (the position when the video stopped). The HMD 200 plays back the video virtual object 205 from the playback position of the video virtual object 205 when the user 10 moves their gaze away from the video virtual object 205 or a predetermined time before the movement of the video virtual object 205. That is, the HMD 200 plays back the video virtual object 205 from a playback position that is temporally before the playback position where the movement of the video virtual object 205 stopped and that is a predetermined time before the user 10 moves their gaze away from the video virtual object 205 or a predetermined time before the user 10 moves their gaze away from the video virtual object 205. This allows the user 10 to view the video that was played back between the time the user 10 moves their gaze away from the video virtual object 205 and the time the user 10 leaves the control area range 206.

[0047] FIG. 13 is a diagram used to explain an example of processing performed by an information processing device when a user approaches within a predetermined distance of a paused video virtual object, based on recorded mutual placement information and information about the user's position and the user's gaze direction after approaching within the predetermined distance. That is, this diagram illustrates an example in which, when playing back a video virtual object, the information processing device plays back the video virtual object from a playback position that is rewinded by the amount of time the user moved from the user's current position within the control area range to the position where the user left the control area range. In FIG. 13, user 10 is at position 207 within control area range 206 and is watching the operation of video virtual object 205. After this, as indicated by arrow 1301, when user 10 moves to position 1302 and leaves control area range 206 without directing his or her gaze at video virtual object 205 for a predetermined time, the operation of video virtual object 205 stops. Then, the user 10 approaches the paused video virtual object 205, and the HMD 200 plays the video virtual object 205 based on the recorded mutual placement information and information about the user's position and the direction of the user's line of sight after approaching within a specified distance. However, if the action of video virtual object 205 is played back from a playback position where user 10 has moved their gaze away from video virtual object 205, the resume playback position may not be the resume playback position that user 10 considers preferable. In such a case, the resume playback position that user 10 considers preferable may be, for example, the playback position when user 10 starts moving from position 207 toward position 1302. Therefore, HMD 200 records the time until user 10 leaves control area range 206 without directing their gaze toward video virtual object 205, and the time of user movement 1303 from the user's stay position to the position where they leave control area range 206. Then, when user 10 moves to position 1302 and leaves control area range 206 without directing their gaze toward video virtual object 205 for a predetermined time, HMD 200 plays back video virtual object 205 from a playback position rewound by the user's movement time. Therefore, when user 10 approaches within a predetermined distance from the paused video virtual object, HMD 200 performs uninterrupted playback based on the recorded mutual placement information and information about the position of user 10 after approaching within the predetermined distance and the direction of line of sight of user 10. Therefore, when the video virtual object is played back, user 10 can view the operation of video virtual object 205 without interruption from the playback position of video virtual object 205 that user 10 was viewing at position 207. The HMD 200 can determine the user's viewing importance of the video virtual object 205 based on information about the time it takes for the user 10 to leave the control area range 206 without directing their gaze at the video virtual object 205. The HMD 200 can then select an appropriate playback position based on that determination. The HMD 200 can perform playback control such as selecting the rewind time, such as whether to play back the video virtual object 205 in a form rewound by the user's movement time, or to play back from the playback position when the user 10 took their gaze off the video virtual object 205.

[0048] Next, an information processing device that, when approaching within a predetermined distance from a paused video virtual object, presents multiple playback position candidates as playback positions for the video virtual object and selects and notifies the user of a playback position based on the viewing importance of the playback position of the video virtual object, will be described with reference to FIG. 14. In FIG. 14, parts shown in FIGS. 1 to 5 and 7 to 13 and assigned the same reference numerals perform the same operations as those already described in those figures, and therefore detailed descriptions thereof will be omitted. As shown in FIG. 14, when user 10 approaches a paused video virtual object and moves into control area range 206, HMD 200 selects multiple playback position candidates, such as user positions 1401, 1402, and 1403, as positions from which to view playback of video virtual object 205. Here, HMD 200 notifies the user by displaying a message image indicating the reason for selecting each playback position candidate and information regarding the playback position of the video virtual object. The HMD 200 generates images of virtual objects such as circles, arrows, and dots that indicate the positional and directional relationships (viewing positions and directions) between the video virtual object and the user's position and line of sight at each playback position candidate, and the gaze point, and superimposes and displays them in three-dimensional space to notify the user. The user 10 then positions and faces one of the selected playback position candidates, or gazes at the gaze point. The HMD 200 then resumes playback of the video virtual object from the playback position of the playback position candidate that corresponds to the viewing position and direction relationship or gaze point. The playback position candidates are selected based on information such as the length and frequency of user 10's gaze, highlight scenes (such as the scene with the most total views, including information about other viewers' viewing), and the relationship between each viewing position and direction. For example, the playback position at the beginning of a scene that user 10 gazed at for the longest time is selected as a candidate, along with information about the relationship between the viewing position and direction. The viewing position and direction information for the playback position candidates is recorded as position and direction information with a predetermined range. By selecting playback position candidates in this way, the user can review important scenes when resuming viewing of the video virtual object, or can resume viewing by omitting unimportant scenes if they run out of time to view.

[0049] Next, an information processing device for a case where a video virtual object is newly fixed and positioned after the user leaves the control area and the operation of the video virtual object is paused will be described with reference to FIG. 15 . In the example of FIG. 15 , the HMD 200 changes the position and orientation of the video virtual object in accordance with the movement of the HMD 200 while maintaining the relative position and orientation between the video virtual object 205 and the position of the user 10 and the line of sight of the user 10 at the time of the pause. Various coordinate systems are used to represent three-dimensional physical objects and virtual objects. The three-dimensional world coordinate system is a coordinate system considered in a world, which is a geometric model of real space and virtual space. The position and orientation of the video virtual object 205 do not change even when the HMD 200 moves. On the other hand, there is a coordinate system of the model itself that is suitable for representing each part of the three-dimensional model, and there is a local coordinate system of the three-dimensional model relative to the world coordinate system. In the local coordinate system of the HMD 200, the video virtual object 205 is fixed and positioned so that the position and direction of the video virtual object 205 change in accordance with the movement of the HMD 200 and follow the movement of the HMD 200. FIG. 15 shows an example of the local coordinate system of the HMD 200. That is, FIG. 15 shows a case where the position and direction of the video virtual object 205 are changed in accordance with the movement of the HMD 200 while maintaining the relative position and direction between the video virtual object 205 and the user 10 or the line of sight of the user 10 at the time of pause. In FIG. 15, parts with the same reference numerals shown in FIGS. 1 to 5 and 7 to 14 have the same operations as those already described in those figures, and therefore detailed description thereof will be omitted.

[0050] 15(a) shows the case where user 10 moves to position 1602 away from control area range 206 as indicated by arrow 1601, pausing the operation of video virtual object 205, and then a predetermined time has elapsed. In this case, HMD 200 changes the position and direction of video virtual object 205 in accordance with the movement of HMD 200, while maintaining the relative position and direction between video virtual object 205 and user 10 or the line of sight of user 10 at the time of the pause. Then, HMD 200 fixes and positions the video virtual object anew, as indicated by video virtual object 1603, and notifies user 10 whether or not to resume the playback operation. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200, allowing the user 10 to select whether to perform a new fixed placement at the position where the user 10 is gazing or whether to resume the playback operation. The HMD 200 then executes a new fixed placement and playback operation of the video virtual object 205 based on the selection result of the user 10 who received the notification. That is, the HMD 200 notifies the user 10 whether to perform a fixed placement and playback of the video virtual object 205 in the local coordinate system of the HMD 200, and controls the fixed placement and playback based on the selection result of the user 10 in response to the notification. Thus, the user 10 can resume viewing the video virtual object 205 while maintaining the relative position and direction between the video virtual object 205 and the user 10 or the user 10's line of sight at the time of the pause. Therefore, even if the user 10 cannot return to the location where the video virtual object 205 was placed, the user 10 can appropriately resume viewing the video virtual object 205.

[0051] Furthermore, when the size of the video virtual object 205 is larger than a predetermined threshold value when the video virtual object is fixedly placed in the local coordinate system of the HMD, the HMD 200 performs the processing described below. That is, as shown in FIG. 15(b), the HMD 200 displays a reduced video virtual object 1611 fixedly placed in the local coordinate system of the HMD 200 and resumes the playback operation of the video virtual object 1611. Similarly, when the size of the video virtual object is larger than a predetermined threshold value when the video virtual object is fixedly placed in the local coordinate system of the HMD, the HMD 200 performs the processing described below. As shown in FIG. 15(c), the HMD 200 displays a video virtual object 1612, which is an excerpt of a predetermined range including the gaze point of the video virtual object, fixedly placed in the local coordinate system of the HMD 200, and resumes the playback operation of the video virtual object 1612. Therefore, the user 10 can resume viewing the playback operation of the video virtual object while maintaining a field of view other than the video virtual object. Thus, viewing can be conveniently stopped and resumed by an operation at a suitable timing according to the user's situation and the arrangement of the three-dimensional video virtual object.

[0052] Next, the fixed placement and playback operation of a video virtual object when a specific viewing location, such as a chair, or another viewer is present within the control area range and the user leaves the control area range will be described with reference to FIG. 15(d). FIG. 15(d) illustrates a case where a specific viewing location, such as a chair 1621, and another viewer 11 are present at position 1624 within the control area range 206. In this case, when a predetermined time has elapsed since the user 10 moved out of the control area range 206, the HMD 200 duplicates the video virtual object while maintaining the positional and directional relationship between the video virtual object and the user's position and line of sight before pausing. The HMD 200 then newly fixes and places the duplicated video virtual object 1623 and notifies the user 10 whether or not to play back the operation of the video virtual object 1623. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200 that allows the user 10 to select whether or not to perform a new fixed placement or resume playback operation by gazing at the screen. Then, the HMD 200 controls the new fixed placement and playback operation of the video virtual object 1623 based on the selection result of the notified user 10 . This allows a user who has left the control area to resume viewing the video virtual object in another location by duplicating the video virtual object and playing it in a new fixed position, while maintaining the position and orientation of the originally placed video virtual object. In other words, while taking into account the situation of the original viewing location of the video virtual object, a user who has left the control area can view the video virtual object in another location before returning to the original placement location. Furthermore, when using an HMD, for example, multiple users wearing HMDs may view a video of a virtual object (hereinafter, sometimes referred to as a video virtual object) together. In such cases, the video virtual object may be placed in a suitable space first, and then viewed by the users. Furthermore, if a user wants to view the video virtual object in the same way as when viewing a physical object, a similar solution may be possible even if the video virtual object is displayed in an obstructive position on the HMD screen.

[0053] Furthermore, when the user returns to the original position after resuming viewing of the video virtual object, the HMD 200 erases the newly fixedly placed video virtual object 1623. This leaves only the original video virtual object 205 in a fixed position, eliminating the confusion caused by unnecessary video virtual objects. Furthermore, if there is a specific viewing location such as a chair but no other viewers, the HMD 200 may perform the process described below. That is, the HMD 200 erases the playback time of the originally placed video virtual object 205 and the newly fixedly placed video virtual object 1623. 623 The playback resumes from a playback position shifted back by the amount of time the video was viewed. This makes it possible to synchronize playback of the originally placed video virtual object with the newly placed video virtual object.

[0054] On the other hand, if a specific viewing location such as a chair or other viewers exists within the control area, and resuming viewing at that location is prioritized, the HMD 200 may be controlled not to fix and place the video virtual object in the local coordinate system of the HMD or to play it back. This prevents the user's actions until they return to the location where the video virtual object is placed from being impeded, allowing the user to quickly return to the location where the video virtual object is placed.

[0055] It should be noted that the settings of each control shown in FIGS. 15(b) to 15(d) may be changed according to the user's preference.

[0056] Next, a case will be described in which, when the user approaches another possible placement location, the video virtual object is newly fixed and placed while maintaining the relative positional and directional relationship between the video virtual object and the user's position and line of sight at the time of pause, and the playback operation is resumed. FIG. 15(e) shows a case in which the user 10 moves away from the control area range 206 to position 1631, and then approaches a round platform 1633, another possible placement location within the control area range 1632, and moves to position 1634. Here, when the user 10 approaches the round platform 1633, another possible placement location, as shown in FIG. 15(f), the HMD 200 performs the processing described below. The HMD 200 identifies whether the video virtual object 1635 can be fixed and placed on the round platform 1633, another possible placement location, while maintaining the relative positional and directional relationship between the video virtual object 205 and the user's position and line of sight at the time of pause. Here, if the HMD 200 identifies that the video virtual object 1635 is possible to be fixed and placed, it notifies the user 10 that the video virtual object 1635 is possible to be fixed and placed. In this notification, the HMD 200 displays a selection screen on the display screen of the HMD 200, allowing the user 10 to select whether to perform a new fixed placement or resume playback operation by gazing at the screen. Then, based on the selection result of the notified user 10, the HMD 200 controls the new fixed placement and playback operation of the video virtual object 1635. Therefore, even if the user changes the placement location of the video virtual object, the user can resume viewing the video virtual object while maintaining the positional and directional relationship between the video virtual object and the user's position and line of sight before pausing. Therefore, viewing can be conveniently stopped and resumed at a timing appropriate for the user's situation. Note that the round table 1633 is an example of another possible placement location, and other examples are also possible. Also, although the case where the round table 1633, which is another possible placement location, is within the control area range 1632 has been shown, it may be outside the control area range 1632. Furthermore, if a specific viewing location such as a chair or another viewer is present within the original control area range, the HMD 200 may be controlled not to notify that a new fixed placement of the video virtual object can be performed at another location, and not to execute fixed placement / playback control. This provides the effect of being able to execute notification instructions and fixed placement / playback control that take into account the situation of the original viewing location of the video virtual object.

[0057] An example of the configuration of an information processing device according to this embodiment will be described with reference to FIG. 16. FIG. 16 is a block diagram showing an example of the configuration of an HMD as an example of the information processing device according to this embodiment. In FIG. 16, parts shown in FIGS. 1, 3-5, and 7-15 and assigned the same reference numerals have the same operations as those already described in those figures. Therefore, some detailed descriptions of those parts may be omitted. In FIG. 16, the HMD 200 is configured appropriately using a left-eye gaze sensor 201, a right-eye gaze sensor 202, a camera 203, a positioning sensor 215, a geomagnetic sensor 216, an acceleration sensor 1504, a gyro sensor 1505, an operation input interface 1507, a display processing device 1508, a processor 1520, a memory 1530 storing a program 1531 and information data 1532, a vibration generating device 1541, an audio input device 1542, an audio output device 1543, and a communication device 1544, and each component is connected to each other via a bus 1550.

[0058] The camera 203 is a device that captures the surrounding field of view in front of the camera, and acquires the camera-captured image by converting light incident from a lens into an electrical signal using an image sensor. The camera 203 is appropriately installed so as to be able to capture the field of view in front of the eyes of the user 10. In a video see-through HMD, the camera 203 captures an actual object in real space, such as a desk 204, and the HMD 200 displays the captured image of the actual object on the display processing device 1508.

[0059] The left eye gaze sensor 201 and the right eye gaze sensor 202 are capable of detecting the movement and direction of the left eye and the right eye, respectively, and capturing the gaze of the user 10. Note that the process of detecting eye movement can utilize well-known techniques that are commonly used as eye tracking processing. For example, in a method using corneal reflex, a technique is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of ​​the reflected light from the infrared LED irradiation (corneal reflex) is used as a reference point, and the eye movement and gaze are detected based on the position of the pupil relative to the position of the corneal reflex.

[0060] An example of the positioning sensor 215 is a Global Positioning System (GPS). This is a device that receives radio signals from artificial satellites to detect the current location, and can identify the current location of the user 10 wearing the HMD 200. As another example, a camera or a ranging sensor (described later) may be used as the positioning sensor. The current location can be determined by analyzing images captured by a camera or three-dimensional point cloud data measured by a ranging sensor, and techniques such as SLAM (Simultaneous Localization and Mapping) are known. Alternatively, a geomagnetic sensor, acceleration sensor, or gyro sensor (described later) may be used as the positioning sensor. The current location can be determined as a relative position from a reference position, and techniques such as PDR (Pedestrian Dead Reckoning) are known. Alternatively, positioning may be performed using Wi-Fi. The current location can be determined by detecting differences in the strength and arrival time of radio waves from multiple Wi-Fi access points and performing triangulation.

[0061] The geomagnetic sensor 216 is a sensor that measures the Earth's magnetic field to detect the direction of the user, and can detect the direction of the user wearing the HMD 200. A three-axis type that detects geomagnetism in the up and down directions in addition to the front-back and left-right directions can be used, and by capturing changes in the geomagnetism in response to the movement of the HMD 200, it is also possible to detect the movement of the HMD 200.

[0062] The acceleration sensor 1504 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, shock, and the like. When the only acceleration applied is gravity, the acceleration sensor 1504 can calculate the tilt angle using the gravity vector and its projection on the acceleration sensor axis, and measure and detect the degree of tilt with respect to the ground. The acceleration sensor 1504 provided in the HMD 200 can detect the tilt of the HMD 200. The gyro sensor 1505 is a sensor that detects the angular velocity in the rotation direction, and can capture the vertical, horizontal, and diagonal orientation states. The gyro sensor 1505 can measure and detect the amount of movement in each direction. Therefore, the orientation of the HMD 200, such as its direction, can be detected using the acceleration sensor 1504 and gyro sensor 1505.

[0063] The ranging sensor 1503 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. For example, a LiDAR (Light Detection and Ranging) sensor can be used, which irradiates an object with laser light such as infrared light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object. A TOF (Time of Flight) sensor can also be used, which measures the distance by measuring the reflection time of pulsed light irradiated on the object for each pixel. A millimeter-wave radar can also be used, which emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The ranging sensor 1503 measures the distance and direction to a physical object such as a desk 204, and the HMD 200 can confirm, for example, that a video virtual object 205 is placed on the desk 204 based on the measurement information.

[0064] The processor 1520 is configured using an appropriate semiconductor device such as a CPU. The processor 1520 controls each component of the HMD 200 and realizes functions of the OS, middleware, applications, and other functions by executing an operating system (OS) 1533 and application programs 1534 for operation control, etc., stored in the memory 1530. The HMD 200 includes a virtual object processing unit 1521, a playback control processing unit 1522, a mutual placement information processing unit 1523, a predetermined gaze range processing unit 1524, a suitable viewing placement processing unit 1525, a playback position candidate processing unit 1526, a surrounding situation processing unit 1527, and a placement feasibility processing unit 1528. These units (1521 to 1528) are program modules executed by the processor 1520, and the HMD 200 controls the operation of each function using these units (1521 to 1528).

[0065] The memory 1530 is configured with a nonvolatile storage device or the like, and stores various programs 1531 and information data 1532 handled by the processor 1520 or the like. The information data 1532 stores virtual object information 1535, mutual arrangement information 1536, predetermined gaze range information 1537, suitable viewing arrangement information 1538, arrangement location surrounding information 1539, and the like.

[0066] In the case of an optical see-through HMD, the display processing device 1508 can be configured to include a projection unit that projects virtual objects and notification information to the user, and a transparent half mirror that displays the projected virtual objects in front of the user's eyes as if they were floating together with real objects in the field of view in front of the user's eyes. In the case of a video see-through HMD, the display processing device 1508 can be configured to include a display such as a liquid crystal panel that displays real objects in front of the user's eyes captured by the camera 203 together with virtual objects, etc. In this way, the user 10 can view the real objects in the field of view image in front of the user's eyes superimposed on the virtual objects, etc.

[0067] The operation input interface 1507 is an input means such as a keyboard, key buttons, or touch keys, and is configured to allow the user 10 to set and input information that he or she wishes to input. The operation input interface 1507 may be provided in a position or form within the HMD 200 that allows the user 10 to easily perform input operations, or may be separate from the main body of the HMD 200 and connected via a wired or wireless connection. The HMD 200 may also display an input operation screen on the display screen of the display processing device 1508 and capture input operation information based on the position on the input operation screen to which the user's line of sight is directed. The HMD 200 may also display a pointer on the input operation screen and capture input operation information input by the user operating the pointer using the operation input interface 1507. The user 10 may also utter a voice indicating an input operation, and the HMD 200 may collect the sound using the voice input device 1542 and capture the input operation information. Furthermore, gestures corresponding to predetermined operations may be registered, and the HMD 200 may acquire the user's operation movements using the camera 203 or the like to capture input operation information.

[0068] The voice input device 1542 is a device that collects external sounds and the user's own voice using a microphone and converts them into voice data. The HMD 200 can take in instruction information from the voice of the user 10 into the HMD 200 and easily execute operations in response to the instruction information.

[0069] The audio output device 1543 outputs audio from a speaker based on the audio data, and can provide audio notification information to the user 10. For example, if the user 10 is located or facing a different position or direction from the viewing position or direction before the pause within the control area range 206, audio is emitted to notify the user that the user's viewing position or direction is different from the viewing position or direction before the pause.

[0070] The vibration generating device 1541 generates vibrations under the control of the processor 1520, and converts into vibrations the notification instruction information to the user 10 transmitted by the HMD 200. The vibration generating device 1541 transmits vibrations to the head of the user wearing the HMD 200, thereby making it possible to notify the user 10 of the notification instruction information, thereby improving usability.

[0071] The communication device 1544 is a communication interface that performs wireless communication with other devices via short-range wireless communication. The communication device 1544 includes a communication processing circuit, an antenna, etc. that correspond to various predetermined communication interfaces, and transmits and receives various information, control signals, etc. It may also include a telephone communication network.

[0072] The timer 1545 measures the time until the user 10 leaves the control area range 206 without directing his / her gaze at the video virtual object 205, the time of the user movement 1303 from the user's stay position to the position where the user leaves the control area range, the time elapsed after the user 10 leaves the control area range 206, etc. The processor 1520 records the time measured by the timer 1545 in the memory 1530.

[0073] The virtual object processing unit 1521 generates a stereoscopic video virtual object 205, such as a soccer game video, and places it in three-dimensional space. It also records information about the position and direction of the placed video virtual object 205 in memory 1530. If the user 10 is outside the control area range 206 for a predetermined time, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 changes the position and direction of the video virtual object in accordance with the movement of the HMD 200 while maintaining the relative position and direction between the video virtual object and the user or the user's line of sight at the time of pause. The virtual object processing unit 1521 then newly fixes and places the video virtual object in the local coordinate system of the HMD 200. When the video virtual object 205 is fixed and placed in the local coordinate system of the HMD 200, if the size of the video virtual object is larger than a predetermined threshold, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 reduces the video virtual object 205, or extracts and arranges an area of ​​a predetermined range including the gaze point for the video virtual object recorded by the gaze predetermined range processing unit 1524 described below. Furthermore, when user 10 re-enters control area range 206 after pausing the video virtual object, virtual object processing unit 1521 performs the processing described below. That is, a virtual object indicating the relative position and direction between video virtual object 205 and user 10 or user 10's line of sight before the recorded video virtual object was paused, and user 10's gaze point 232 relative to video virtual object 205 is generated and displayed. For example, virtual object processing unit 1521 generates images of virtual objects such as circles, arrows, and dots indicating the above information and displays them superimposed on three-dimensional space. Also, virtual object images of message images indicating the reason for selecting a playback position candidate for the video virtual object and information regarding the playback position of the video virtual object are generated and displayed superimposed on three-dimensional space. Furthermore, virtual object images such as circles, arrows, and dots indicating information regarding the position and direction (each viewing position and direction) relationship between the video virtual object and the user at the playback position candidate are generated and displayed superimposed on three-dimensional space.

[0074] When the user 10 moves a predetermined distance away from the video virtual object 205, the playback control processing unit 1522 pauses the operation of the video virtual object 205. Then, when the user 10 approaches within a predetermined distance from the video virtual object 205, the playback control processing unit 1522 performs the processing described below. That is, the playback operation of the paused video virtual object 205 is controlled based on the recorded mutual placement information, information regarding the user's position and the user's line of sight after approaching within the predetermined distance, information regarding a candidate playback position for the video virtual object 205, and the like. Examples of controlling the playback operation include playing the video virtual object 205 as is, or rotating the video virtual object 205 and playing it. Furthermore, the video virtual object 205 may be rewound a predetermined time and played back based on information regarding the user's viewing importance for the video virtual object and the viewing importance of the playback position of the video virtual object. The playback control processing unit 1522 also controls the playback and stop operation of the video virtual object 205 fixedly positioned in the local coordinate system of the HMD 200.

[0075] The mutual placement information processing unit 1523 acquires the relative position and direction relationship between the video virtual object and the user's position or line of sight before stopping the video virtual object, generates mutual placement information, and records it in the memory 1530. For example, as the mutual placement information, the position and direction of the user 10 or the user's line of sight relative to the video virtual object 205 at the stay position where the user 10 stayed before leaving the control area range 206 is recorded. Here, as an example, the stay position is the starting position of the movement of the user 10 when the user 10 moves outside the control area range 206. This stay position may be a position where the change in sensor value is smaller than a predetermined value based on information from various sensors (e.g., the positioning sensor 1503, the acceleration sensor 1504, and the gyro sensor 1505) for a predetermined time, and it is determined that the user 10 is not moving. Furthermore, if the user 10 moves outside the control area range 206 without facing the video virtual object 205 for a predetermined time, the mutual placement information processing unit 1523 performs the processing described below. That is, the mutual arrangement information processing unit 1523 records the position where the user 10 stayed before leaving the control area range 206 and the direction from that position towards the video virtual object 205 . Here, the direction from the stay position toward the video virtual object 205 can be acquired as appropriate. As an example, the orientation of the HMD 200 displaying the video virtual object 205 before pausing may be detected by the acceleration sensor 1504 or the gyro sensor 1505, and the detected orientation of the HMD 200 may be recorded as the direction from the stay position toward the video virtual object 205. The direction of a virtual object or the like that is superimposed may also be taken into consideration. In this way, the mutual placement information processing unit 1523 records the mutual placement information using information about the position of the user 10 relative to the video virtual object 205 and the line of sight of the user 10 that was acquired and recorded before the video virtual object 205 was paused. That is, the mutual placement information processing unit 1523 selects and acquires the viewing position and direction of the user 10 relative to the video virtual object 205 before pausing from the information, and records it as the mutual placement information. The mutual placement information may also be recorded as position and direction information having a predetermined range.

[0076] The predetermined gaze range processing unit 1524 selects a predetermined gaze range 233 centered on the user's gaze position 232 before stopping, which was identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, generates predetermined gaze range information, and records it in memory 1530. For example, when the gaze position of the user 10 relative to the video virtual object 205 does not move within a predetermined time, the predetermined gaze range processing unit 1524 identifies a location including the stationary gaze position as the gaze position 232. Alternatively, when the amount of movement of the gaze position of the user 10 relative to the video virtual object 205 within a predetermined time is equal to or less than a predetermined threshold, the predetermined gaze range processing unit 1524 identifies a location including a position where the gaze movement is equal to or less than the predetermined threshold as the gaze position 232.

[0077] The preferred viewing arrangement processing unit 1525 selects the preferred viewing position and direction of the video virtual object as preferred viewing arrangement information and records the selected preferred viewing arrangement information in the memory 1530. For example, the preferred viewing arrangement information may include the user's position and gaze point before pausing. It may also include the preferred viewing position and direction relative to the view of a virtual object, etc., that the user gazed at in the video scene of the video virtual object at the time of pausing. It may also include the preferred viewing position and direction relative to a specific virtual object, which the user gazed at in the video scene of the video virtual object at the time of pausing, or which the user specified in advance. Here, the gaze point and the gazed virtual object, etc., may be acquired, for example, based on the left eye gaze sensor 201 and the right eye gaze sensor 202. The direction relative to the specific virtual object may be estimated by tracking the specific virtual object. The user's viewing position is a position where the target gaze point, etc., can be viewed optimally, and may be estimated, for example, based on the position of each virtual object.

[0078] The playback position candidate processing unit 1526 selects candidates for the video playback position at which the video virtual object is rewound and played, and stores the candidates in memory 1530. The candidates for the video playback position may be selected based on information such as the length and frequency of user 10's gaze, highlight scenes (such as the scene with the most total views, including information about the viewing of other viewers), etc. The candidates for the video playback position may also be selected based on information such as the relationship between the video virtual object at the playback position candidate and the user's position or the user's line of sight (viewing position and direction). In addition to such viewing conditions, candidates may also be selected based on the content of the video virtual object (for example, information about a highlight tag indicating a highlight assigned to the video virtual object, or information about the action of a target that is focused on in the highlight).

[0079] Surrounding situation processing unit 1527 records an image within the control area range captured by camera 203 or displayed on display processing device 1508 in memory 1530 as placement location surrounding information 1539. Then, based on the recorded information, surrounding situation processing unit 1527 determines whether a specific viewing location such as a chair or other viewers exists around the video virtual object (within the control area range). Note that the objects to be determined as specific viewing locations such as chairs or other viewers may be controlled so that they can be applied not only to real objects and people but also to virtual objects other than the video virtual object.

[0080] The placement possibility processing unit 1528 analyzes the distance measurement results from the distance measurement sensor 1503 and the image captured by the camera 203 or displayed on the display processing device 1508. The placement possibility processing unit 1528 then identifies the distance from the user 10 to the real object or another virtual object, the size of the real object or another virtual object, the presence or absence of a horizontal surface, and the type (for example, a desk or a table), and determines whether the video virtual object can be placed.

[0081] With the above configuration, the HMD 200 performs the following basic operation. When the user moves a predetermined distance away from the stereoscopic video virtual object 205 arranged in three-dimensional space by the virtual object processing unit 1521, the HMD 200 causes the playback control processing unit 1522 to pause the operation of the video virtual object. Furthermore, the mutual placement information processing unit 1523 acquires and records the relative position and direction relationship between the video virtual object and the user or the user's line of sight before the pause as mutual placement information. After the pause, the user approaches within a predetermined distance from the video virtual object. At this time, the playback control processing unit 1522 controls the playback operation of the video virtual object based on the recorded mutual placement information, information about the user's position and the user's line of sight after approaching within the predetermined distance, information about a candidate playback position for the video virtual object 205, and the like. The playback operation can be controlled by playing back the video virtual object as is, rotating the video virtual object, or rewinding a predetermined time before playing back.

[0082] Furthermore, in the HMD 200, the predetermined gaze range processing unit 1524 selects a predetermined gaze range centered on the user's gaze point in the state before the pause, generates and records it as predetermined gaze range information, and after the pause, if it is determined that the user's line of sight has entered the predetermined gaze range indicated by the predetermined gaze range information, the HMD 200 resumes the playback operation of the video virtual object.

[0083] In addition, in the HMD 200, when the operation of the video virtual object is stopped, the preferred viewing position / direction of the video virtual object is selected by the preferred viewing position processing unit 1525 as preferred viewing position information, and the selected preferred viewing position information is notified to the user. The preferred viewing position information includes the user's position and gaze point before pausing, and a preferred viewing position / direction for a scene such as a virtual object that the user was gazing at in the video scene of the video virtual object at the time of pausing. Also included is a preferred viewing position / direction for a specific virtual object that the user was gazing at in the video scene of the video virtual object at the time of pausing, or that was specified in advance.

[0084] In addition, the HMD 200 uses the playback position candidate processing unit 1526 to record the video playback position at which the video virtual object is rewound and played, as well as candidate viewing positions and directions at those video playback positions. The playback position candidates are selected and recorded based on gaze time information, such as the length and frequency of the user's gaze on the video virtual object, highlight scenes (such as the scene with the most total views, including information about the viewing of other viewers), and information about each viewing position and direction. That is, the playback position candidates are selected and recorded based on the viewing status of such video virtual objects and the contents of the video virtual objects (for example, information about highlight tags indicating highlights assigned to the video virtual object, and information about the behavior of the target focused on in the highlights). The playback operation of the video virtual object is resumed from the video playback position selected by the user from the selected and recorded playback position candidates.

[0085] Furthermore, the user moves a predetermined distance away from a stereoscopic video virtual object 205, such as a soccer game video, arranged in three-dimensional space by the virtual object processing unit 1521. Then, the playback control processing unit 1522 pauses the operation of the video virtual object, and the mutual placement information processing unit 1523 acquires and records the relative position and direction relationship between the video virtual object and the user's position and line of sight before the pause as mutual placement information. If the user remains a predetermined distance away from the video virtual object for a predetermined period of time, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 fixes and places the position and direction of the video virtual object in the local coordinate system of the HMD based on the mutual placement information recorded by the mutual placement information processing unit 1523 so that the position and direction of the video virtual object follow the movement of the HMD. Furthermore, the playback control processing unit 1522 controls the playback and stop operation of the video virtual object.

[0086] Furthermore, if the size of the video virtual object is larger than a predetermined threshold when the video virtual object is fixedly placed in the local coordinate system of the HMD by the virtual object processing unit 1521, the HMD 200 performs the following process: The video virtual object is reduced in size, or a predetermined range of the area including the gaze point for the video virtual object recorded by the predetermined gaze range processing unit 1524 is extracted and fixedly placed in the local coordinate system of the HMD.

[0087] Furthermore, if the surrounding situation processing unit 1527 determines that a specific viewing location or other viewers exists around the placed video virtual object, the HMD 200 performs the following processing: That is, the virtual object processing unit 1521 does not fix and place the video virtual object in the local coordinate system of the HMD, or controls the video virtual object to be duplicated and fixed and placed in the local coordinate system of the HMD.

[0088] Furthermore, the user moves a predetermined distance away from a stereoscopic video virtual object 205, such as a soccer game video, arranged in three-dimensional space by the virtual object processing unit 1521. Then, the playback control processing unit 1522 pauses the operation of the video virtual object, and the mutual placement information processing unit 1523 acquires and records the relative position and direction relationship between the video virtual object and the user's position and line of sight before the pause as mutual placement information. If the placement possibility processing unit 1528 subsequently determines that a new video virtual object can be placed when the user approaches another possible placement location, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 controls the video virtual object to be newly fixed and placed while maintaining the relative position and direction relationship between the video virtual object and the user's position and line of sight at the time of the pause, and resumes its playback operation.

[0089] Thus, an information processing device can be realized that, when viewing a three-dimensional video virtual object placed in a specific location, can easily and conveniently stop or resume viewing by operating at an appropriate timing according to the user's situation and the placement of the three-dimensional video virtual object without requiring any special operation by the user. For example, even if an emergency arises while a user wearing an HMD is viewing, the operation of the video virtual object can be stopped or resumed at an appropriate timing in an easy-to-use manner.

[0090] In the above explanation, an HMD has been used as a specific example of an information processing device, but the present invention applies to all devices with similar functions, and it goes without saying that the same functions and effects can be obtained with information processing devices other than HMDs, such as smartphones and smartwatches.

[0091] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another example, or to add the configuration of another example to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each example with other configurations.

[0092] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0093] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software by a processor interpreting and executing a program that implements each function. The processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0094] The range and shape of the control area may be freely set by the user. Alternatively, the user may select an appropriate range from a plurality of pre-registered patterns. Information about the range of the control area determined by the user is stored in memory 1530 and used for processing by the HMD 200. [Explanation of symbols]

[0095] 10: User, 200: Head-mounted display (HMD), 201: Left eye gaze sensor, 202: Right eye gaze sensor, 203: Camera, 204: Real object (desk), 205: Video virtual object, 206: Control area range, 208: Display screen, 215: Positioning sensor, 216: Geomagnetic sensor, 1503: Distance measurement sensor, 1504: Acceleration sensor, 1505: Gyro sensor, 1507: Operation input interface, 1508: Display processing device, 1520: Processor, 1521: virtual object processing unit, 1522: playback control processing unit, 1523: mutual placement information processing unit, 1524: predetermined gaze range processing unit, 1525: suitable viewing placement processing unit, 1526: playback position candidate processing unit, 1527: surrounding situation processing unit, 1528: placement possibility processing unit, 1530: memory, 1541: vibration generating device, 1542: audio input device, 1543: audio output device, 1544: communication device, 1545: timer, 1550: bus

Claims

1. An information processing device worn by a user, a processor for controlling the playback of a stereoscopic video virtual object; a positioning sensor for detecting the position of the user; a gaze sensor for detecting a gaze direction of the user; a memory; The processor: Recording user information in the memory, the user information including at least one of a placement position where the reproduced video virtual object is placed and the position of the user detected by the positioning sensor and a line of sight direction of the user detected by the line of sight sensor; When the positioning sensor detects that the user has moved out of a range that is a predetermined distance from the placement position, control is performed to stop the playback of the video virtual object; When the user enters the range after stopping the playback of the video virtual object, control is performed to resume the playback of the video virtual object based on the user information recorded in the memory and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor.

1. An information processing device comprising:

2. 2. The information processing device according to claim 1, When the user enters the range after stopping the playback of the video virtual object, if the position detected by the positioning sensor and the gaze direction detected by the gaze sensor correspond to the user information recorded in the memory, the processor controls to resume the playback of the video virtual object.

1. An information processing device comprising:

3. 2. The information processing device according to claim 1, When the user enters the range after stopping the playback of the video virtual object, if the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the processor controls to issue a notification to the user.

1. An information processing device comprising:

4. 2. The information processing device according to claim 1, The processor: When the amount of change in the user's gaze direction detected by the gaze sensor is equal to or less than a threshold value within a predetermined time, the range of the video virtual object in the gaze direction is recorded in the memory as a gaze range, When the user enters the range after stopping the playback of the video virtual object and the direction of the user's line of sight enters the gaze range, control is performed so that playback of the video virtual object is resumed.

1. An information processing device comprising:

5. 2. The information processing device according to claim 1, The processor: A position and a direction suitable for viewing the video virtual object are recorded in the memory as suitable placement information; When the user enters the range after stopping the reproduction of the video virtual object, the preferable placement information recorded in the memory is notified to the user.

1. An information processing device comprising:

6. The information processing device according to claim 1 comprises: a display that displays the video virtual object reproduced by the processor; When the user enters the range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor differs from the user information recorded in the memory, the processor controls the video virtual object to be rotated and displayed on the display.

1. An information processing device comprising:

7. The information processing device according to claim 1 comprises: Equipped with a display, The processor: Controlling the display so that the video virtual object is superimposed on a real object, When the user enters the range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor differs from the user information recorded in the memory, the video virtual object is rotated according to the shape or size of the actual object and displayed on the display.

1. An information processing device comprising:

8. 2. The information processing device according to claim 1, When the user enters the range after stopping playback of the video virtual object, the processor controls the video virtual object to resume playback by returning it to a playback time position that is a predetermined time before the playback time position at which playback of the video virtual object was stopped.

1. An information processing device comprising:

9. 2. The information processing device according to claim 1, The processor: recording a plurality of playback positions in the memory as spatial positions at which the video virtual object is played back; When the user enters the range after stopping the playback of the video virtual object, the user is notified of the plurality of playback positions in space; Controlling the playback of the video virtual object so as to resume from the playback position selected by the user; 1. An information processing device comprising:

10. 2. The information processing device according to claim 1, The processor: When the user does not enter the range even after a predetermined time has elapsed since the playback of the video virtual object was stopped, the control is performed so that the user is notified of whether or not to resume the playback of the video virtual object; When the user selects to resume playback, the video virtual object whose playback has been resumed is controlled to be placed at a position different from the placement position recorded in the memory.

1. An information processing device comprising:

11. An information processing device that has a function of generating and displaying a stereoscopic video virtual object and is worn by a user, A processor and a storage unit are included, The processor: Recording mutual arrangement information, which is a mutual positional and directional relationship between the video virtual object and the user's position and line of sight, in the storage unit; When the user moves away from the video virtual object and the distance between the video virtual object and the user becomes equal to or greater than a predetermined distance, Controlling to stop playback of the video virtual object; After a predetermined time has elapsed since the stop, the video virtual object is controlled to be positioned so as to follow the movement of the information processing device in a state based on the mutual positioning information recorded in the storage unit, Controlling the playback operation of the placed video virtual object.

1. An information processing device comprising:

12. The information processing device according to claim 11, The processor: Controlling to notify the user whether or not to reproduce the action of the placed video virtual object; When the user who received the notification selects to reproduce the action of the placed video virtual object, Controlling to execute a playback operation of the arranged video virtual object.

1. An information processing device comprising:

13. The information processing device according to claim 11, The processor: When it is determined that the size of the video virtual object is larger than a predetermined threshold, the size of the video virtual object is reduced and the video virtual object is placed.

1. An information processing device comprising:

14. The information processing device according to claim 11, The processor: A gaze point where the user gazed on the video virtual object before the stop is recorded in the storage unit, When it is determined that the size of the video virtual object is larger than a predetermined threshold, the video virtual object is controlled to be placed in a predetermined area including the gaze point recorded in the storage unit.

1. An information processing device comprising:

15. An information processing method for a device worn by a user, Playing 3D virtual objects, The reproduced video virtual object is arranged and displayed so as to be superimposed on a real object; Recording user information including at least one of a placement position where the video virtual object is placed, a position of the user, and a line of sight direction of the user; When it is detected that the user has moved out of a range that is a predetermined distance from the placement position, playback of the video virtual object is stopped; After stopping the playback of the video virtual object, when the user enters the range, the playback of the video virtual object is resumed based on the recorded user information. An information processing method comprising:

Images