Information processing device and information processing method

Abstract

This invention provides an information processing device and information processing method that allow for easy and convenient stopping and resuming of viewing of three-dimensional virtual image objects. [Solution] An information processing device and information processing method used by a user, which reproduces a three-dimensional virtual image object, displays the reproduced virtual image object superimposed on a physical object, and records user information including the placement position of the virtual image object and at least one of the user's position and line of sight. When it is detected that the user has moved outside a predetermined distance range from the placement position, the reproduction of the virtual image object is stopped. After the reproduction of the virtual image object is stopped, when the user enters the above range, the reproduction of the virtual image object is resumed based on the recorded user information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] In recent years, Augmented Reality (AR) technology that attaches digital information to the real world and reflects and expands virtual objects (virtual objects) on a virtual space created by CG (Computer Graphics) in the real space has been widely used. As a device using this technology, an information processing apparatus that can easily handle virtual objects while recognizing the real space in three dimensions has become popular. And as an example of this, a Head Mounted Display (HMD) having a display unit, a camera unit, etc. mounted on the head can be cited.

[0003] As a method for pausing and playing the operation of a video of a virtual object (hereinafter sometimes referred to as a video virtual object), Patent Document 1 has the following description. That is, Patent Document 1 states that "the playback unit 107 plays back the moving image data stored in the storage unit 104 (301). The composite reality processing unit 103 acquires the viewpoint position (or the position of the user) and the line-of-sight direction of the user by the position and orientation estimation result by the self-position and orientation estimation unit 106 and the line-of-sight tracking function in parallel with the playback processing by the playback unit 107 (S302). The composite reality processing unit 103 calculates the deviation Φ of the line of sight from the video playback screen G and the distance D between the user's own position and the video playback screen G. The playback unit 107 adjusts the playback speed according to the deviation Φ and the distance D (S303). For example, when the deviation Φ is deviated by a predetermined angle and the distance D is equal to or greater than a predetermined value, the playback unit 107 slows down or stops the playback speed. Note that the conditions for the deviation Φ and the distance D may be at least either one. This is a criterion for determining whether the user is looking at the video playback screen G."

[0004] Furthermore, Patent Document 2 states that "the user terminal 1500 has a function to acquire measurement data measuring the position and orientation of the user terminal 1500, and places the subject object and virtual camera in a virtual 3D space, and controls the position and orientation of the virtual camera in conjunction with the measurement data. The user terminal 1500 then generates a virtual space image captured by the virtual camera and displays it as a monitor image, while recording camera work data 550 that can reproduce the position and orientation of the virtual camera." and "the playback image generation control unit 232 performs control to generate a playback image, which is an image of the virtual 3D space from the virtual camera at a given playback timing of the camera work, based on the camera work data 550 recorded by the recording control unit 224." [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] International Publication Number WO2021 / 132168 A1 [Patent Document 2] Japanese Patent Publication No. 2020-119334 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] As mentioned above, Patent Document 1 shows how to stop and play a virtual video object screen based on the position, direction, and distance of the user's gaze relative to a positioned planar virtual video object. On the other hand, it does not consider the case of a three-dimensional virtual video object that can be viewed from multiple positions and directions. Therefore, there is a problem in stopping and resuming viewing of a three-dimensional virtual video object in a suitable state and timing.

[0007] As mentioned above, Patent Document 2 describes controlling the playback of a virtual video object based on information regarding camera work (camera position and orientation). However, it does not provide any indication regarding the user's gaze toward the virtual video object or the distance from the user. Therefore, a problem arises in that it is not possible to control the stopping and playback of the virtual video object based on the position, direction, and distance of the user's gaze toward the virtual video object. Furthermore, although Patent Document 2 describes suitable techniques for "video creation," it does not describe suitable "viewing," and there is a problem in stopping and resuming "viewing" at an appropriate timing. Therefore, the present invention aims to provide an information processing device and an information processing method that allow for easy and convenient stopping and resuming of viewing of three-dimensional virtual image objects, taking the above problems into consideration. [Means for solving the problem]

[0008] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0009] (1) An information processing device to be worn by a user, comprising a processor that controls the playback of a three-dimensional virtual image object, a positioning sensor that detects the user's position, a gaze sensor that detects the user's gaze direction, and a memory. The processor records user information in the memory, including the placement position where the played-out virtual image object is placed, and at least one of the user's position detected by the positioning sensor and the user's gaze direction detected by the gaze sensor. The processor also controls the playback of the virtual image object to stop when the positioning sensor detects that the user has moved out of a predetermined distance from the placement position, and when the user enters the above range after the playback of the virtual image object has stopped, it controls the playback of the virtual image object to resume based on the user information recorded in the memory and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor.

[0010] (2) The information processing device described in (1), wherein the processor restarts playback of the video virtual object when the user enters the above range after the playback of the video virtual object has been stopped, and the position detected by the positioning sensor and the direction of gaze detected by the gaze sensor correspond to the user information recorded in memory.

[0011] (3) The information processing device described in (1), wherein the processor controls the device to issue a notification to the user if, after stopping the playback of the video virtual object, the user enters the above range and the position detected by the positioning sensor or the direction of gaze detected by the gaze sensor differs from the user information recorded in memory.

[0012] (4) The information processing device described in (1), wherein the processor records the range of the virtual video object in the direction of the gaze as the gaze range in memory if the amount of change in the user's gaze direction detected by the gaze sensor is below a threshold for a predetermined period of time, and resumes playback of the virtual video object when the user enters the above range and the user's gaze direction enters the gaze range after playback of the virtual video object has been stopped.

[0013] (5) The information processing device described in (1), wherein the processor records in memory a suitable position and orientation for viewing a virtual video object as preferred placement information, and when the user enters the above range after playback of the virtual video object has been stopped, the processor notifies the user of the preferred placement information recorded in memory.

[0014] (6) The information processing device described in (1) includes a display that shows a video virtual object played back by the processor, and the processor controls the rotation of the video virtual object on the display when the user enters the above range after stopping the playback of the video virtual object and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in memory.

[0015] (7) The information processing device described in (1) is equipped with a display, and the processor controls the display so that a virtual image object is superimposed on a physical object, and after the playback of the virtual image object is stopped, if the user enters the above range and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in memory, the processor controls the display to rotate the virtual image object according to the shape or size of the physical object.

[0016] (8)(1) The information processing device described above, wherein the processor controls the system so that when the user enters the above range after playback of the video virtual object has been stopped, it returns to a position a predetermined time before the position where playback of the video virtual object was stopped and then resumes playback of the video virtual object.

[0017] (9)(1) The information processing device described above, wherein the processor records multiple playback positions in memory as positions for playing back a video virtual object, and when the user enters the above range after playback of the video virtual object has been stopped, it notifies the user of the multiple playback positions and controls the device to resume playback of the video virtual object from the playback position selected by the user.

[0018] The information processing device described in (10)(1) is configured such that, if the user does not enter the above range even after a predetermined time has elapsed since the playback of the video virtual object was stopped, the processor notifies the user whether or not to resume playback of the video virtual object, and when the user chooses to resume playback, the processor controls the playback to place the video virtual object at a location different from the location recorded in memory.

[0019] (11) An information processing device that is worn by a user and has the function of generating and displaying a three-dimensional virtual image object, comprising a processor and a memory unit, wherein the processor records mutual positional information in the memory unit, which is the relative positional relationship between the virtual image object and the user's position and line of sight, stops the playback of the virtual image object when the user moves away from the virtual image object and the distance between the virtual image object and the user exceeds a predetermined distance, and after a predetermined elapsed time from the stop, positions the virtual image object in a state based on the mutual positional information recorded in the memory unit so as to follow the movement of the information processing device, and controls the playback operation of the positioned virtual image object.

[0020] The information processing device described in (12)(11), wherein the processor notifies the user whether or not to play back the operation of the placed virtual video object, and if the user who has received the notification chooses to play back the operation of the placed virtual video object, the processor executes the playback operation of the placed virtual video object.

[0021] (13)(11) The information processing device described above, wherein the processor, when it identifies that the size of the virtual video object is larger than a predetermined threshold, reduces the size of the virtual video object and arranges it.

[0022] The information processing apparatus described in (14)(11), wherein the processor records in a storage unit the points of focus that the user was focusing on with respect to the virtual video object before it is stopped, and if it identifies that the size of the virtual video object is larger than a predetermined threshold, it arranges a virtual video object that is an excerpt of a predetermined area including the points of focus recorded in the storage unit.

[0023] (15) An information processing method for a device worn and used by a user, which plays a stereoscopic video virtual object, arranges and displays the played video virtual object so as to superimpose it on a physical object, records user information including at least one of the arrangement position where the video virtual object is arranged, the position of the user, and the line-of-sight direction, and stops playing the video virtual object when it is detected that the user has moved outside a range that is a predetermined distance from the arrangement position. After stopping the playback of the video virtual object, when the user enters the above range, the playback of the video virtual object is resumed based on the recorded user information.

Effect of the Invention

[0024] By using the technology of the present invention, there is an effect that an information processing device and an information processing method can be realized that can easily and conveniently stop and resume viewing of a stereoscopic video virtual object. In addition, problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0025] [Figure 1] This is an example of a diagram schematically explaining the appearance of an embodiment of the information processing device according to the present embodiment. [Figure 2] This is an example of a diagram schematically explaining the specific display screen of the embodiment described in FIG. 1. [Figure 3] This is an example of a diagram explaining the control operation of the information processing device according to the present embodiment shown in FIG. 1. [Figure 4(a)] This is another example of a diagram explaining the control operation of the information processing device according to the present embodiment shown in FIG. 1. [Figure 4(b)] This is an example of a diagram schematically explaining the display screen during the control operation of the information processing device according to the present embodiment shown in FIG. 4(a). [Figure 5(a)] This is another example of a diagram explaining the control operation of the information processing device according to the present embodiment shown in FIG. 1. [Figure 5(b)]This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 6] This is an example flowchart illustrating the basic operation of the information processing device according to this embodiment. [Figure 7] Figure 1 is an example of a diagram illustrating the recording operation of the information processing device according to this embodiment. [Figure 8] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 9] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 10(a)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 10(b)] Figure 10(a) is an example of a diagram illustrating the control operation results of the information processing device according to this embodiment. [Figure 11] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 12] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 13] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 14] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(a)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(b)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(c)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(d)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(e)]This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 15(f)] This is another example of a diagram illustrating the control operation of the information processing device according to this embodiment, as shown in Figure 1. [Figure 16] This block diagram shows an example of an HMD configuration as an example of an information processing device according to this embodiment. [Modes for carrying out the invention]

[0026] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. As a specific example of the information processing device according to this embodiment, a head-mounted display (HMD) that is worn on the user's head and displays and allows viewing of information in real space and virtual space will be described as an example. Figures 1, 3, 4(a), 5(a), 5(b), and 7 are examples of diagrams that schematically show the external appearance of the information processing device according to this embodiment, and Figures 2 and 4(b) are examples of diagrams that explain the display screen within the field of view of the information processing device according to this embodiment. In Figure 1, an HMD 200 is worn on the head of user 10. This HMD 200 includes a left eye gaze sensor 201 and a right eye gaze sensor 202 that detect the gaze of the user 10's left and right eyes, a camera 203 that photographs the outside world, a positioning sensor 215 that detects the position on Earth, and a geomagnetic sensor 216 that detects the direction the HMD is facing.

[0027] Unlike an optical see-through HMD in which user 10 directly views real objects in the field of view in front of them, a video see-through HMD uses a camera 203 to capture images of real objects in the field of view in front of them and displays the captured images of the real objects on a display. In addition to real objects, the HMD 200 generates three-dimensional virtual image objects and displays them in 3D within user 10's field of view. As a result, user 10 can view the virtual image objects as if they were real objects present in the real world. Furthermore, the HMD200 records information regarding the position and orientation of the placed three-dimensional virtual video object. Figure 1 shows the case where a virtual video object 205, which is a video of a soccer game, is placed in the space on a physical object, a desk 204. Although not shown in the figure, it is preferable to display a selection screen for the user 10 to select a physical object on which to place the virtual video object, allowing the user 10 to place it in any location. In addition, the HMD200 detects the user 10's position using the equipped positioning sensor 215 and detects the direction the user 10 is facing using the equipped geomagnetic sensor 216.

[0028] When user 10 is at a position 207 inside the control area range 206, which is within a predetermined distance from the virtual image object 205 as shown in Figure 1, user 10 can see the virtual image object 205. That is, as shown in Figure 2, user 10 can see an image of a soccer game, which is an example of a virtual image object 205 placed on a desk 204, on the display screen 208 within the field of view. Note that the desk 204 may be a different virtual object from the virtual image object 205. At this time, that is, when displaying the virtual image object 205, the HMD 200 can grasp the position and orientation of the virtual image object 205 placed in three-dimensional space. In addition, the HMD 200 can detect user 10's position and orientation using the positioning sensor 215 and the geomagnetic sensor 216, and user 10's gaze using the left eye gaze sensor 201 and the right eye gaze sensor 202. Therefore, the HMD200 can identify and acquire the relationship between the position and orientation of the virtual video object 205 and the user 10, as well as the relationship between the direction in which the virtual video object 205 is displayed and the user 10's line of sight.

[0029] Here, user 10 moves to a position 209 outside the control area range 206, a predetermined distance away from the virtual video object 205, as indicated by arrow 210 in Figure 3. At this time, the HMD 200 temporarily pauses the operation of the virtual video object 205, which is a video of a soccer game in progress. This allows the operation of the virtual video object to be immediately paused simply by user 10 moving outside the control area range 206, without requiring any special operation from the user. The HMD 200 also identifies the relationship between the position and orientation of the virtual video object 205 and user 10, as well as the relationship between the direction in which the virtual video object 205 is displayed and user 10's line of sight, and records the relative position and orientation relationship before the pause as relative position information. Here, the HMD200 records the following information as relative positioning information. For example, the HMD200 records the relative position and orientation relationship of the virtual video object 205 at the location where user 10 stayed before leaving the control area range 206 (the starting position of movement when user 10 moves outside the control area range 206). Furthermore, if user 10 moves outside the control area range 206 without facing the direction of the virtual video object 205 for a predetermined period of time, the HMD200 records the following information as relative positioning information. For example, the HMD200 records the location where user 10 stayed before leaving the control area range 206 and the direction from that location toward the virtual video object 205. In this way, the HMD200 extracts and acquires the viewing position and direction of the user 10 relative to the virtual video object 205 before pausing, from the information regarding the user 10's position or gaze direction relative to the virtual video object 205 recorded before pausing. The HMD200 then records the viewing position and direction of the user 10 relative to the virtual video object 205 before pausing as relative position information. For example, the dwelling position can be the position where the user 10 stays while viewing the virtual video object, that is, the position where the user 10 continues to view the virtual video object 205 without moving for a predetermined time. Furthermore, the relative position information may be recorded as position and direction information with a predetermined range. This allows it to be recorded as information indicating a rough position and direction. Alternatively, instead of recording this as mutual placement information, or in addition to recording it, the placement location of the virtual video object 205 and the user's position and gaze direction may be recorded as user information.

[0030] After the operation of the virtual video object 205 is paused, user 10 approaches the virtual video object 205 as indicated by arrow 211 in Figure 3. At this time, if user 10 approaches within a predetermined distance from the virtual video object 205 and is positioned and facing the same viewing position and direction as before the pause, the HMD 200 resumes playback of the virtual video object 205 of the soccer game. In other words, if user 10 is positioned within a predetermined range based on the recorded relative positioning information and facing the same direction as in the recorded relative positioning information, the HMD 200 resumes playback of the virtual video object 205. That is, the HMD 200 determines whether user 10 is in the same position and facing the same direction as before the pause, based on the relative positioning information recorded in the state before the pause and information regarding user 10's position and gaze direction after approaching within a predetermined distance.

[0031] This allows the playback of the virtual video object 205 to be immediately resumed without requiring any special operation from the user 10. Furthermore, the user 10 can resume viewing and playback of the virtual video object 205 with a roughly reproduced viewing state from before the user stopped the virtual video object 205. Thus, the HMD 200 can reduce the difference in the image perceived by the user 10 before stopping and after playback, which occurs due to differences in the user 10's viewing direction, for the virtual video object 205, which looks different depending on the viewing angle. As a result, the user 10 can continuously view the virtual video object 205.

[0032] On the other hand, after the operation of the virtual video object 205 is paused, user 10 approaches the virtual video object 205 from a different position and direction than the viewing position and direction of the virtual video object 205 before the pause, as shown by arrow 221 in Figure 4(a). When user 10 reaches position 222, HMD 200 creates and displays a message 223 on the display screen 208 within the field of view, as shown in Figure 4(b), indicating that user 10's viewing position and direction is different from that of user 10 before the pause, and notifies user 10. Here, HMD 200 can determine whether or not the viewing position and direction is different from that before the pause based on the relative position information recorded in the state before the pause. Figure 4(b) shows the display screen 208 within the field of view of the HMD 200, and shows an example of a message 223 to be displayed on the display screen 208, in which case a message with the content "Notification message: Viewing position and direction are different..." is generated and displayed. This allows the HMD 200 to notify the user 10 that the user 10's re-entry position and direction into the control area range 206 is different from the viewing position and direction of the virtual video object 205 before pausing. In addition, the HMD 200 can notify the user 10 that, for the virtual video object 205 which looks different depending on the angle, there is a difference in the user's perceived image before pausing and after playback due to the difference in viewing direction. The form of notification is not particularly limited and may be a message, as shown in Figure 4(b), a symbol indicating the meaning, or a warning via sound or vibration. The HMD200 may, for example, display a line indicating the control area range 206 as a thick or solid line, change the color of the displayed line, or make the line blink for a predetermined time to provide notification.

[0033] Upon receiving such a notification, user 10 can recognize that they are in a different position or facing a different direction than their viewing position before pausing. User 10 can then choose whether to resume playback of the virtual video object 205 as is, or to perform rotation or other control on the virtual video object 205 before resuming playback. If user 10 chooses to resume playback of the virtual video object 205 as is, the user can resume viewing the virtual video object 205 from their desired viewing position and direction.

[0034] When user 10 moves a predetermined distance from the virtual video object 205 and outside the control area range 206, as indicated by arrow 231 in Figure 5(a), and the HMD 200 stops the operation of the virtual video object 205, information regarding user 10's gaze is recorded. Specifically, as shown in Figure 5(a), the HMD 200 extracts a predetermined gaze range 233 centered on the point of gaze 232 of user 10 before stopping, which was identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, and records it as predetermined gaze range information. If user 10's gaze towards the virtual video object 205 does not move for a predetermined period of time, the HMD 200 identifies the area including the position at the end of the gaze as the point of gaze 232. Alternatively, if the amount of movement of user 10's gaze towards the virtual video object 205 is below a predetermined threshold, the HMD 200 identifies the area including the position at the end of the gaze as the point of gaze 232.

[0035] After pausing, user 10 moves into the control area range 206 as indicated by arrow 234 in Figure 5(b). The HMD 200 then identifies the position and orientation corresponding to the recorded relative positioning information, and when it identifies that user 10's gaze point 235 has entered the predetermined gaze range 233 indicated by the predetermined gaze range information, it resumes playback of the virtual video object 205. As a result, the HMD 200 can resume playback of the virtual video object 205 with user 10's gaze point 235 directed towards the vicinity of the gaze point 232 that user 10 was looking at when the virtual video object 205 stopped. Therefore, even after playback resumes, user 10 can gaze at almost the same area as before the pause, from the viewing position and direction before the pause, and can smoothly view the resumed virtual video object 205 without any discomfort from before the pause. Furthermore, the HMD200 can resume playback of the virtual video object 205 if the state in which user 10 was paying particular attention to and watching the virtual video object 205 before the stoppage is reproduced. As shown in Figures 5(a) and 5(b), the virtual video object 205 is a soccer game, and for example, user 10 focuses on the area around the soccer ball while watching. In this case, even when playback is resumed after pausing, user 10 can continue watching without interruption, focusing on the area around the soccer ball they are focusing on.

[0036] Furthermore, when user 10 re-enters the control area range 206 after pausing the video virtual object 205, the HMD 200 may display the following information. Specifically, the HMD 200 may be controlled to display the relative position and orientation relationship between the recorded video virtual object 205 before pausing and the position of the re-entering user 10 and the user 10's line of sight. The manner in which the relative position and orientation relationship is displayed is not particularly limited as long as the relationship can be understood. For example, the HMD 200 may display the difference between the position of the virtual object 205 before pausing and the current position of the HMD 200 as a numerical value. If this difference becomes smaller than a predetermined amount, the HMD 200 may notify the user of this fact by displaying text information or outputting sound. For example, the HMD 200 may display a compass rose indicating the direction in which the video virtual object 205 was displayed before pausing, and a compass rose indicating the current orientation of the HMD 200. Furthermore, the HMD200 may be controlled to generate images of virtual objects such as circles, arrows, or dots that indicate the user 10's gaze point 232 relative to the virtual video object 205, and to superimpose these images onto the three-dimensional space. As a result, when user 10 re-enters the control area range 206, user 10 can immediately grasp the relative position and direction relationship between the virtual video object 205 and user 10's position before the pause, as well as user 10's line of sight. Furthermore, user 10 can immediately grasp the user's gaze point 232 on the virtual video object 205. Thus, when viewing a three-dimensional virtual video object 205 placed in a specific location, user 10 can easily and conveniently stop and resume viewing with actions at an appropriate timing depending on the situation, without requiring any special operation from user 10.

[0037] Next, the basic operation of the information processing device according to this embodiment will be explained using a flowchart. Figure 6 is an example of a flowchart explaining the basic operation of the information processing device according to this embodiment. The basic operation of the information processing device (in this example, HMD200) is appropriately executed using hardware resources described later. In Figure 6, the virtual video object 205 is played back and displayed on the HMD200 worn by the user 10 (S701). When the user 10 moves a predetermined distance away from the virtual video object 205 and outside the control area range 206 (S702), the HMD200 pauses the playback of the virtual video object 205 (S703). At this time, the HMD200 records the relative position and direction relationship between the virtual video object 205 and the user 10's position and the user 10's line of sight before the pause as relative position information. The HMD200 also records the predetermined gaze range centered on the gaze point before the pause as gaze predetermined range information (S704). After this, with the operation of the virtual video object 205 temporarily suspended, user 10 approaches the virtual video object 205 and enters the control area range 206 within a predetermined distance (S705). Here, the HMD200 identifies whether user 10 is in the viewing position before stopping and whether user 10 is facing the direction before stopping (i.e., whether user 10 is located within a predetermined range based on the recorded mutual positioning information and whether user 10 is facing the predetermined direction). Furthermore, the HMD200 identifies whether user 10's gaze is directed inside the predetermined gaze range (S706). If the HMD200 identifies that user 10 is in the viewing position before stopping and is facing the direction before stopping, and furthermore, that user 10's gaze is directed inside the predetermined gaze range, the HMD200 resumes playback of the video virtual object 205 (S707). Furthermore, if the display and playback of the video virtual object 205 is not yet finished, the HMD200 returns to sequence S702 and repeats this flow. When the HMD200 has finished displaying and playing the video virtual object 205, it terminates this flow (S708).

[0038] On the other hand, if user 10 is in the viewing position before stopping and is not facing the same direction as before stopping, HMD 200 notifies user 10 that they are in a different position and direction (S709). If user 10, upon receiving the notification, decides to resume playback of the video virtual object 205 (S710), user 10 inputs this to HMD 200 using an appropriate method, and HMD 200 resumes playback of the video virtual object 205 (S707). The input method by user 10 may include, for example, button operation or gestures. At this time, HMD 200 may resume playback of the video virtual object 205 as is, or it may rotate the video virtual object 205 to a viewing-friendly orientation before resuming playback. If user 10 decides not to resume playback of the video virtual object 205 (S710), and has not finished displaying and playing the video virtual object 205, the HMD200 returns to sequence S702 and repeats this flow. When the display and playback of the video virtual object 205 is finished, the HMD200 terminates this flow (S708). Through the above operations, an information processing device (HMD200 in this example) is provided that allows the user to easily and conveniently stop and resume viewing of video virtual objects without requiring any special operation from the user.

[0039] Next, the information processing device used to notify the user of their position and gaze point before pausing, and the optimal viewing position and direction of the virtual video object at the time of pausing, will be explained using Figures 7, 8, and 9. Note that in Figures 7, 8, and 9, the parts shown in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b) and denoted by the same reference numerals have the same operation as those already explained in Figures 1, 2, 3, 4(a), 4(b), 5(a), and 5(b), so their detailed explanations will be partially omitted. In Figure 7, the HMD 200 can capture the position of the user 10 using the positioning sensor 215, and can capture the gaze point on the virtual video object 205 that the user 10 is looking at using the left eye gaze sensor 201 and the right eye gaze sensor 202. Therefore, by notifying the user of the position and gaze point of the user 10, which were captured before the pause, as viewing-optimal positioning information, the HMD200 enables the user 10 to understand their viewing position before the virtual video object 205 stopped.

[0040] Next, Figures 8 and 9 will be explained. Note that parts shown in Figure 7 and denoted by the same reference numerals have the same operation as those already explained in Figure 7, so some detailed explanations of them will be omitted. In Figure 8, when user 10 moves outside the control area range 206 as indicated by arrow 210, HMD 200 pauses the operation of the video virtual object 205, which is a video of a soccer match. At this time, HMD 200 selects the position and direction of the most suitable points of interest in the video scene within the paused video virtual object 205 as viewing-optimal placement information and notifies user 10. Here, the position and direction of the most suitable points of interest in the video scene is the position and direction in which the virtual object or other scenery that the user was focusing on in the video scene can be seen up close without any obstructions. In the case of the video of a soccer match shown as the video virtual object 205 in Figure 8, as an example, let's assume that the user was focusing on the soccer ball 801. Therefore, the HMD200 selects a suitable viewing position and direction 802 as viewing-optimal arrangement information that allows the user to view the soccer ball they were focusing on as if they were holding it in their hands, without it being obstructed by other virtual objects, and notifies the user 10 of this. Thus, the HMD 200 can suggest to user 10 a suitable viewing position and direction for a certain virtual object within the virtual video object 205 that user 10 was focusing on. Furthermore, user 10 can re-enter this suggested viewing position and direction and resume playback of the virtual video object 205 at a suitable point of view. In another example, the HMD 200 could suggest to the user a suitable viewing position and direction for a certain soccer player within the virtual video object 205 that the user was focusing on, allowing them to view the player from the front or at an angle rather than from behind, and without being obstructed by other virtual objects. Therefore, playback can be resumed at a suitable point of view.

[0041] In Figure 9, when user 10 moves outside the control area range 206 as indicated by arrow 210, the HMD 200 pauses the movement of the video virtual object 205, which is a video of a soccer match. At this time, the HMD 200 selects a suitable viewing position and direction for a specific virtual object within the video scene of the paused video virtual object 205 as viewing preference position information and notifies the user. In Figure 9, using the video of a soccer match shown as the video virtual object 205, let's say, for example, that a certain soccer player 901 is selected as the specific virtual object. The HMD 200 extracts the soccer player 901, enclosed in a rectangular frame 902, as the specific virtual object. Then, the HMD 200 selects a suitable viewing position and direction 903, which allows the movement of the specific virtual object, the soccer player 901, to be viewed as if it were right in front of the user's eyes without being obstructed by other virtual objects, as viewing preference position information and notifies the user 10. Thus, in the video virtual object 205 that user 10 is viewing, the HMD 200 can suggest to user 10 a position and direction in which virtual objects such as virtual objects specified in advance by user 10 or important people can be viewed suitably, regardless of user 10's point of focus. Furthermore, user 10 can re-enter this suggested viewing position and direction and resume playback of the video virtual object 205 at a suitable point of interest. In selecting the aforementioned specific virtual object or the viewing position and direction in which the specific virtual object can be viewed suitably, the HMD 200 may make a decision as described below. That is, the HMD 200 may record highlight scenes that are frequently viewed by the viewer or specific virtual objects that the viewer frequently looks at in relation to the soccer video, which is the video virtual object 205, and make a decision based on this record.

[0042] As another example, the user 10 may specify a particular virtual object while viewing the video virtual object 205, and the HMD 200 may be controlled to select a suitable viewing position and direction for the specified virtual object as viewing-optimal positioning information and notify the user 10. In this way, the HMD 200 can notify the user 10 of a suitable viewing position and direction for the specific virtual object arbitrarily selected by the user 10.

[0043] Next, the information processing device used when performing attitude control, such as rotating the virtual video object, during the resumption of playback after the virtual video object has been stopped will be explained using Figures 10(a), 10(b), 11, and 12. In Figures 10(a), 10(b), 11, and 12, the parts shown in Figures 1-9 and denoted by the same reference numerals have the same operations as those already explained in Figures 1-9, so their detailed explanations will be partially omitted. In Figure 10(a), when the user 10 moves outside the control area range 206 as indicated by the arrow 210, the HMD 200 temporarily pauses the operation of the virtual video object 205, which is a video of a soccer game. The HMD 200 also records the relative position and direction relationship between the virtual video object 205 and the user 10, as well as the user 10's line of sight, as relative position information before the virtual video object 205 is stopped. After the pause, the user 10 approaches the virtual video object 205 from direction 1001, within a predetermined distance. Here, the HMD200 rotates the virtual video object 205 according to the direction in which the user 10 approaches 1001 and the position after approaching 1002, based on the recorded position of the user 10 when stopped and the position and direction of the user 10's line of sight. That is, as shown in Figure 10(a), the user 10 approaches the virtual video object 205, which is the soccer game video, from a position 1002 on the opposite side in approximately the same direction 1001 as the direction in which the user 10 is away 210. In this case, the HMD200 rotates the soccer game video by approximately 180 degrees, as indicated by arrows 1003 and 1004. As a result, as user 10 approaches, they can view the soccer match video in the same arrangement as before it was stopped, as shown in Figure 10(b). That is, both before stopping and after resuming playback, the players of the team wearing black uniforms can be seen on the left side of the field of view, and the players of the team wearing white uniforms can be seen on the right side of the field of view, allowing for smooth viewing without any sense of incongruity even when transitioning from stopping to resuming playback. User 10 approaches the video virtual object 205 from a direction different from the viewing direction of user 10 before the video virtual object 205 was stopped, or after approaching, user 10 is positioned in a direction different from the viewing direction of user 10 before the video virtual object 205 was stopped. Even in such cases, user 10 can resume viewing the video virtual object 205 with the general viewing state that user 10 had before the video virtual object 205 was stopped reproduced, without user 10 having to move to the viewing position and direction before it was stopped.

[0044] Figure 11 shows the case where, after pausing, user 10 approaches the video virtual object 205, which is a video of a soccer game, from a direction 1101 that is almost perpendicular to the direction 210 away from user 10. Here, based on the recorded position and direction relationship of user 10 and user 10's line of sight at the time of pausing, we consider rotating the video virtual object 205 by approximately 90 degrees in the direction indicated by arrows 1102 and 1103 according to the direction of approach 1101 of user 10. In this case, as shown in Figure 11, the video of the soccer game 1104 rotated by approximately 90 degrees will extend beyond the top of the desk 204. Therefore, if the video virtual object 205 is not contained within the desk 204 or other placement location due to its size and shape, the HMD 200 will not rotate the video virtual object 205. This prevents the occurrence of a discrepancy between the video virtual object 205 and its placement location due to the rotation of the video virtual object 205.

[0045] Figure 12 shows the case where, after pausing, user 10 approaches the virtual video object 205 from direction 1001, similar to the case shown in Figure 10(a). Here, if there is a specific viewing location such as a chair 1201 inside the control area range 206 that encompasses the virtual video object 205, the HMD 200 will not rotate the soccer game video, which is the virtual video object 205. In this case, since there is a high possibility that viewing of the virtual video object 205 will resume from sitting in a chair or from a specific viewing location, the HMD 200 can enable viewing without causing any particular discomfort by deliberately not rotating the virtual video object 205. Furthermore, if there are other viewers, not rotating is also beneficial in order to avoid affecting their viewing. Thus, the effect is obtained in which playback control of the virtual video object 205 and notification instructions to user 10 can be made while considering the situation of the viewing location of the virtual video object 205.

[0046] Next, we will explain the information processing device for playing back a virtual video object by rewinding its playback position (playback time position) using Figures 7 and 13. This information processing device processes information based on recorded relative positioning information, information on the playback position of the virtual video object 205, and information on the user's position and the direction of the user's gaze after approaching the paused virtual video object, when the user approaches within a predetermined distance. Note that in Figure 13, the parts shown in Figures 1-5 and 7-12 and denoted by the same reference numerals have the same operation as those already explained in those figures, so some detailed explanations of them will be omitted. When user 10 is within the control area range 206, as shown in Figure 7, user 10's gaze is directed towards the virtual video object 205. If user 10 then attempts to move outside the control area range 206, user 10's gaze will move away from the virtual video object 205. At this point, the HMD 200 records the playback position of the virtual video object 205 when user 10 looks away. If user 10 moves further outside the control area range 206, the operation of the virtual video object 205 stops. Therefore, when user 10 approaches the temporarily stopped virtual video object 205, the HMD 200 performs the processing described below based on the recorded relative positioning information, information regarding the playback position of the virtual video object 205, and information regarding the user's position and the direction of their gaze after approaching within a predetermined distance. That is, when the HMD 200 resumes playback of the virtual video object 205's operation based on this information, it does not resume playback from the playback position where the virtual video object 205's operation stopped (the position where the video stopped). The HMD200 plays the virtual video object 205 from a playback position that was set to a predetermined time prior to when the user 10 looked away from the virtual video object 205. In other words, the HMD200 plays the virtual video object 205 from a playback position that was set to a predetermined time prior to when the user 10 looked away from the virtual video object 205, or when the user 10 looked away from the virtual video object 205. This allows the user 10 to view the video that was played between the time the user 10 looked away from the virtual video object 205 and when the user 10 left the control area range 206.

[0047] Figure 13 illustrates an example of processing performed by an information processing device when a user approaches a paused virtual video object within a predetermined distance, based on recorded relative positioning information and information regarding the user's position and gaze direction after approaching the predetermined distance. Specifically, it illustrates an example where, when playing back the virtual video object, the information processing device plays back the virtual video object from a playback position that is set back by the amount of time the user traveled from their current position within the control area to their exit from the control area. In Figure 13, user 10 is at position 207 within the control area 206 and is watching the operation of the virtual video object 205. Subsequently, as indicated by arrow 1301, when user 10 moves to position 1302 and exits the control area 206 without looking at the virtual video object 205 for a predetermined time, the operation of the virtual video object 205 stops. Then, when user 10 approaches the paused video virtual object 205, the HMD 200 plays back the video virtual object 205 based on the recorded relative positioning information and information about the user's position and the direction of the user's gaze after approaching within a predetermined distance. However, if the playback of the virtual video object 205's actions is resumed from a playback position where user 10 has taken their eyes off the virtual video object 205, the restart playback position may not be a restart playback position that user 10 considers preferable. In such a case, one example of a restart playback position that user 10 might consider preferable is the playback position when user 10 begins moving from position 207 towards position 1302. Therefore, the HMD 200 records the time until user 10 leaves the control area range 206 without looking at the virtual video object 205, and the time of user movement 1303 from the user's current position to the position where they leave the control area range 206. Then, if user 10 moves to position 1302 and leaves the control area range 206 without looking at the virtual video object 205 for a predetermined period of time, the HMD 200 resumes playback from the playback position of the virtual video object 205, rewinding by the amount of user movement time. Therefore, when user 10 approaches a paused virtual video object within a predetermined distance, the HMD 200 performs uninterrupted playback based on the recorded relative positioning information and information regarding user 10's position and gaze direction after approaching the predetermined distance. As a result, when playing back the virtual video object, user 10 can view the operation of the virtual video object 205 without interruption from the playback position of the virtual video object 205 that user 10 was viewing at position 207. Furthermore, the HMD200 can determine the user's viewing importance of the virtual video object 205 based on information regarding the time it takes for user 10 to leave the control area range 206 without looking at the virtual video object 205. Based on this determination, the HMD200 may then select an appropriate playback position. The HMD200 can perform playback control, such as selecting the rewind time, to play back the virtual video object 205 by the amount of time the user moved, or to play it back from the position where user 10 took their eyes off the virtual video object 205.

[0048] Next, using Figure 14, we will explain an information processing device that, when approaching a paused video virtual object within a predetermined distance, lists multiple playback position candidates for the video virtual object, selects a playback position according to the importance of viewing the playback position of the video virtual object, and notifies the user. In Figure 14, the parts shown in Figures 1-5 and 7-13 with the same reference numerals have the same operation as those already explained in those figures, so some detailed explanations of them will be omitted. As shown in Figure 14, when user 10 approaches a video virtual object that is inactive and moves within the control area range 206, HMD 200 selects multiple playback position candidates, such as user positions 1401, 1402, and 1403, as positions for viewing the playback of the video virtual object 205. Here, HMD 200 notifies the user by displaying a message image that shows the reason for selecting each playback position candidate and information about the playback position of the video virtual object. The HMD200 generates images of virtual objects such as circles, arrows, and dots that indicate the positional and directional relationships (each viewing position and direction) between the virtual video object at each playback position candidate and the user's position and gaze, as well as the point of focus, and displays them superimposed in 3D space for notification. Subsequently, user 10 positions themselves and faces one of the selected playback position candidates, or focuses on the point of focus. When this happens, the HMD200 resumes playback of the virtual video object from the playback position candidate corresponding to that viewing position and directional relationship and the point of focus. The playback position candidates are selected based on information such as the length and frequency of user 10's attention, highlight scenes (including information about other viewers' viewing, such as the scene with the highest total number of views), and the relationship between each viewing position and direction. For example, the beginning of a scene that user 10 spent a long time watching may be listed as a candidate, along with information about the relationship between the viewing position and direction. The viewing position and direction information for the playback position candidates is recorded as position and direction information within a predetermined range. By selecting playback position candidates in this way, the user can review important scenes when resuming viewing of the virtual video object, or, if they can no longer dedicate a long time to viewing, they can resume viewing by omitting less important scenes.

[0049] Next, using Figure 15, we will explain the information processing device for when the user moves out of the control area range, the movement of the virtual video object is temporarily paused, and the virtual video object is then newly fixed in place. In the example in Figure 15, the HMD 200 maintains the relative positional and directional relationship between the virtual video object 205 and the user 10's position and the user 10's line of sight at the time of pause, and changes the position and direction of the virtual video object in accordance with the movement of the HMD 200. Various coordinate systems are used to represent three-dimensional physical objects and virtual objects. The three-dimensional world coordinate system is a coordinate system considered within a geometrically modeled representation of real or virtual space, and the position and direction of the virtual video object 205 do not change even when the HMD 200 moves. On the other hand, there is a coordinate system of the model itself that is suitable for representing each part of the three-dimensional model, and there is a local coordinate system of the three-dimensional model in relation to the world coordinate system. In the local coordinate system of the HMD200, the virtual video object 205 is fixed and positioned so that its position and orientation change in accordance with the movement of the HMD200, thereby following the movement of the HMD200. Figure 15 shows an example of the local coordinate system of the HMD200. Specifically, Figure 15 shows the case where the position and orientation of the virtual video object 205 are changed in accordance with the movement of the HMD200, while the relative position and orientation relationship between the virtual video object 205 and the user 10 and the user 10's line of sight is maintained when the display is paused. In Figure 15, the parts shown in Figures 1-5 and Figures 7-14 and denoted by the same reference numerals have the same operation as the operation already described in those figures, so some detailed explanations of them are omitted.

[0050] Figure 15(a) shows the case where user 10 moves away from the control area range 206 to position 1602 as indicated by arrow 1601, and after a predetermined time has elapsed since the operation of the virtual video object 205 was temporarily suspended. In this case, the HMD 200 changes the position and orientation of the virtual video object 205 in accordance with the movement of the HMD 200, while maintaining the relative position and orientation relationship between the virtual video object 205 and user 10 and user 10's line of sight at the time of suspension. The HMD 200 then fixes the virtual video object in a new position as shown in virtual video object 1603 and notifies user 10 whether or not to resume its playback operation. In this notification, the HMD200 displays a selection screen on its display screen that allows the user 10 to choose whether to create a new fixed position for the virtual video object 205 at the position the user 10 is looking at, or whether to resume playback. Based on the user 10's selection, the HMD200 then performs the new fixed positioning or playback operation for the virtual video object 205. In other words, the HMD200 notifies the user 10 whether to fix the virtual video object 205 in the HMD200's local coordinate system and play it, and controls the fixed positioning or playback based on the user 10's selection in response to the notification. Thus, while the relative position and direction relationship between the virtual video object 205 and the user 10 and the user 10's line of sight is maintained, the user 10 can resume viewing the virtual video object 205. Therefore, even if the user 10 cannot return to the location where the virtual video object 205 was placed, the user 10 can appropriately resume viewing the virtual video object 205.

[0051] Furthermore, when fixing a virtual video object in the local coordinate system of the HMD, if the size of the virtual video object 205 is larger than a predetermined threshold, the HMD 200 performs the following process. That is, as shown in Figure 15(b), the HMD 200 fixes and displays a reduced virtual video object 1611 in the local coordinate system of the HMD 200 and resumes playback of the virtual video object 1611. Similarly, when fixing a virtual video object in the local coordinate system of the HMD, if the size of the virtual video object is larger than a predetermined threshold, the HMD 200 performs the following process. As shown in Figure 15(c), the HMD 200 fixes and displays a virtual video object 1612, which is an excerpt of a predetermined range including the point of focus of the virtual video object, in the local coordinate system of the HMD 200 and resumes playback of the virtual video object 1612. Therefore, the user 10 can resume viewing the playback of the virtual video object while maintaining a field of view other than the virtual video object. Therefore, viewing can be conveniently paused and resumed at an appropriate timing according to the user's situation and the arrangement of the three-dimensional virtual video objects.

[0052] Next, using Figure 15(d), we will explain the fixed placement and playback operation of the virtual video object when the user moves out of the control area range while a specific viewing location such as a chair or other viewers are present within the control area range. Figure 15(d) shows that a specific viewing location, such as a chair 1621, and other viewers 11 are located at position 1624 within the control area range 206. In this case, after a predetermined amount of time has elapsed since the user 10 moved outside the control area range 206, the HMD 200 duplicates the virtual video object while maintaining the positional and directional relationship between the virtual video object before the pause and the user's position and gaze. The HMD 200 then fixes the duplicated virtual video object 1623 in place and notifies the user 10 whether or not to resume the operation of the virtual video object 1623. At this time, the HMD 200 displays a selection screen on its display screen that allows the user 10 to choose whether or not to fixate on the object and resume playback. Then, based on the selection made by the notified user 10, the HMD200 controls the new fixed placement and playback operation of the video virtual object 1623. This allows users who have left the control area to resume viewing the virtual video object in a different location while maintaining the original position and orientation of the virtual video object, by duplicating the virtual video object and repositioning it in a fixed location for playback. In other words, while taking into account the conditions of the original viewing location of the virtual video object, users who have left the control area can view the virtual video object in a different location before returning to the original location. Furthermore, when using an HMD, for example, multiple users, each wearing an HMD, may simultaneously view images of virtual objects (hereinafter sometimes referred to as "visual virtual objects"). In such cases, it is conceivable that the visual virtual object would first be placed in a suitable space for the users to view. Similarly, if a user wants to view a visual virtual object in the same way they would view a real object, or if the visual virtual object is displayed in an inconvenient position on the HMD screen, similar solutions can be considered.

[0053] Furthermore, if the user returns to their original position after resuming viewing of the video virtual object, the HMD200 deletes the newly fixed video virtual object 1623. This eliminates the confusion caused by unnecessary video virtual objects, leaving only the original video virtual object 205 in a fixed position. In addition, if a specific viewing location such as a chair exists but there are no other viewers, the HMD200 may perform the following process. That is, the HMD200 resumes playback from a position shifted backward by the amount of time the original video virtual object 205 was viewed relative to the newly fixed video virtual object 1632. This allows the original video virtual object and the newly fixed video virtual object to be played in sync.

[0054] On the other hand, if a specific viewing location, such as a chair, or other viewers exist within the control area, and priority is given to resuming viewing at that location, the HMD200 may be controlled so that the virtual video object is not fixedly placed in the HMD's local coordinate system and playback is not performed. This avoids interfering with the user's actions until they return to the location where the virtual video object is placed, allowing the user to return to the location of the virtual video object more quickly.

[0055] Furthermore, the controls shown in Figures 15(b) to 15(d), etc., should be made adjustable according to the user's preferences.

[0056] Next, we will explain the case where, when the user approaches another possible placement location, the virtual video object is newly fixed in place while maintaining the relative position and orientation relationship between the virtual video object at the time of pause and the user's position and line of sight, and the playback operation is resumed. Figure 15(e) shows the case where user 10 moves away from the control area range 206 to position 1631, then approaches another possible placement location within the control area range 1632, the round platform 1633, and moves to position 1634. Here, when user 10 approaches the round platform 1633, the other possible placement location, the HMD 200 performs the process described below, as shown in Figure 15(f). The HMD 200 identifies whether it is possible to fix the virtual video object 1635 on the round platform 1633, the other possible placement location, while maintaining the relative position and orientation relationship between the virtual video object 205 at the time of pause and the user's position and line of sight. Here, if the HMD 200 identifies that it is possible to fix the object in place, it notifies user 10 that it is possible to fix the object in place. In this notification, the HMD200 displays a selection screen on its display screen that allows user 10 to choose whether or not to fixate on the virtual video object 1635 or resume playback. Based on user 10's selection, the HMD200 controls the new fixed placement and playback of the virtual video object 1635. Therefore, even if the user changes the placement of the virtual video object, viewing of the virtual video object can be resumed while maintaining the positional and directional relationship between the virtual video object and the user's position and gaze before the pause. Thus, viewing can be conveniently stopped and resumed with timing appropriate to the user's situation. Note that the round stand 1633 is just one example of another possible placement location, and other examples may also be used. Also, although the example shows the round stand 1633 being within the control area range 1632, it may also be outside the control area range 1632. Furthermore, if a specific viewing location such as a chair or other viewers exist within the original control area range, the HMD 200 may be controlled not to notify that a new fixed placement of the video virtual object can be made in another location, and not to execute fixed placement / playback control. This has the effect of being able to execute notification instructions and fixed placement / playback control that take into account the situation of the original viewing location of the video virtual object.

[0057] An example of the configuration of the information processing device according to this embodiment will be described with reference to Figure 16. Figure 16 is a block diagram showing an example of the configuration of an HMD as an example of the information processing device according to this embodiment. In Figure 16, the parts shown in Figures 1, 3-5, and 7-15 and denoted by the same reference numerals have the same operation as those already described in those figures. Therefore, some detailed explanations of them may be omitted. In Figure 16, the HMD 200 is configured using a left eye gaze sensor 201, a right eye gaze sensor 202, a camera 203, a positioning sensor 215, a geomagnetic sensor 216, an acceleration sensor 1504, a gyroscope sensor 1505, an operation input interface 1507, a display processing device 1508, a processor 1520, a memory 1530 that stores a program 1531 and information data 1532, a vibration generating device 1541, an audio input device 1542, an audio output device 1543, and a communication device 1544 as appropriate, and each component is interconnected via a bus 1550.

[0058] Camera 203 is a device that captures the field of view in front of the camera, and acquires a camera image by converting light incident from the lens into an electrical signal with an image sensor. Camera 203 is appropriately positioned to capture the field of view in front of the user 10. In a video see-through type HMD, camera 203 captures real objects in the real space, such as a desk 204, and the HMD 200 displays the captured images of the real objects on the display processing device 1508.

[0059] The left eye gaze sensor 201 and the right eye gaze sensor 202 can detect the movement and direction of the left and right eyes, respectively, and capture the user's gaze direction. The process for detecting eye movement can be carried out using well-known techniques commonly used as eye tracking. For example, in a method using corneal reflection, an infrared LED (Light Emitting Diode) is shone on the face and captured with an infrared camera. The position of the reflected light on the cornea created by the infrared LED irradiation (corneal reflection) is used as a reference point, and the movement of the eye and gaze direction are detected based on the position of the pupil relative to the position of the corneal reflection.

[0060] One example of a positioning sensor 215 is a GPS (Global Positioning System). This device receives radio signals from satellites to detect the current location and can pinpoint the current location of the user 10 wearing the HMD200. Other examples include using a camera or a distance measuring sensor (described later) as a positioning sensor. The current location can be determined by analyzing images captured by a camera or three-dimensional point cloud data measured by a distance measuring sensor, and technologies such as SLAM (Simultaneous Localization and Mapping) are known. Alternatively, a geomagnetic sensor, acceleration sensor, or gyroscope sensor (described later) may be used as a positioning sensor. The current location can be determined as a relative position from a reference location, and technologies such as PDR (Pedestrian Dead Reckoning) are known. Positioning can also be performed using Wi-Fi. The current location can be determined by detecting differences in the strength and arrival time of radio waves from multiple Wi-Fi access points and performing triplicate positioning.

[0061] The geomagnetic sensor 216 measures the Earth's magnetic field and detects the direction it is facing, allowing it to detect the direction the user wearing the HMD200 is facing. By using a 3-axis type sensor that detects the geomagnetic field in the vertical direction in addition to the forward / backward and left / right directions, it is also possible to detect the movement of the HMD200 by capturing the changes in the geomagnetic field in response to the movement of the HMD200.

[0062] The accelerometer 1504 is a sensor that detects acceleration, which is the change in velocity per unit time, and can capture motion, vibration, shock, etc. When the only applied acceleration is gravity, the accelerometer 1504 can determine the tilt angle using the gravity vector and its projection onto the axis of the accelerometer, and measure and detect how much it is tilted relative to the ground. The accelerometer 1504 installed inside the HMD200 can detect the tilt of the HMD200. In addition, the gyro sensor 1505 is a sensor that detects angular velocity in the rotational direction, and can capture the vertical, horizontal, and diagonal orientation. The gyro sensor 1505 can measure and detect how much it has moved in which direction. Therefore, the orientation and other attitudes of the HMD200 can be detected using the accelerometer 1504 and the gyro sensor 1505.

[0063] The distance measuring sensor 1503 is a sensor that can measure the distance and angle to an object and capture the shape of an object as a three-dimensional object. For example, LiDAR (Light Detection and Ranging) is used, which irradiates an object with laser light such as infrared light, measures the scattered light that reflects back, and analyzes and detects the distance to an object at a distance and the state of that object. TOF (Time Of Flight) sensors, which measure the reflection time of pulsed light irradiated onto a subject to measure the distance for each pixel, may also be used. Millimeter-wave radar, which emits millimeter-wave radio waves and captures the reflected waves to detect the distance to the object reflecting the light and the state of the object, may also be used. The distance measuring sensor 1503 measures the distance and direction to a physical object such as a desk 204, and the HMD 200 can use this measurement information to confirm, for example, that a virtual image object 205 is placed on the desk 204.

[0064] The processor 1520 is configured using appropriate semiconductor devices such as a CPU. The processor 1520 controls each component of the HMD200 by executing the operating system (OS) 1533 and application programs 1534 for operation control, etc., stored in the memory 1530, thereby realizing the functions of the OS, middleware, applications, and other functions. The HMD200 includes a virtual object processing unit 1521, a playback control processing unit 1522, a mutual arrangement information processing unit 1523, a gaze predetermined range processing unit 1524, a viewing suitable arrangement processing unit 1525, a playback position candidate processing unit 1526, an ambient situation processing unit 1527, and an arrangement feasibility processing unit 1528. These (1521 to 1528) are program modules executed by the processor 1520, and the HMD200 controls the operation of each function using these (1521 to 1528).

[0065] The memory 1530 is composed of a non-volatile storage device and stores various programs 1531 and information data 1532 handled by the processor 1520 and the like. The information data 1532 includes virtual object information 1535, mutual arrangement information 1536, predetermined viewing range information 1537, viewing-friendly arrangement information 1538, and surrounding information of the arrangement location 1539.

[0066] In the case of an optical see-through HMD, the display processing device 1508 can be configured to include a projection unit that projects virtual objects and notification information to the user, and a transparent half-mirror that forms an image of the projected virtual objects in front of the user's eyes. This allows the user to see both the projected virtual objects and the real objects in front of their eyes as if they were floating together. In the case of a video see-through HMD, the device can be configured to include a display such as an LCD panel that displays both the real objects in front of the user's eyes, captured by the camera 203, and the virtual objects. This allows the user 10 to see the real objects and virtual objects superimposed in the image of their field of view.

[0067] The operation input interface 1507 is an input means such as a keyboard, key buttons, or touch keys, and is configured for the user 10 to input the information they wish to enter. The operation input interface 1507 can be located in a position and form that makes it easy for the user 10 to perform input operations within the HMD 200, or it may be separate from the main body of the HMD 200 and connected by wire or wireless. The HMD 200 may also display the input operation screen on the display screen of the display processing device 1508 and capture input operation information based on the position on the input operation screen where the user's gaze is directed. The HMD 200 may also display a pointer on the input operation screen and capture input operation information entered by the user operating the pointer using the operation input interface 1507. Alternatively, the user 10 may speak to indicate an input operation, and the HMD 200 may collect the sound using the voice input device 1542 to capture the input operation information. Furthermore, gestures corresponding to predetermined operations may be registered, and the HMD200 may acquire user operation actions using the camera 203 or the like to capture input operation information.

[0068] The voice input device 1542 is a device that collects external voices or the user's own voice using a microphone and converts them into voice data. The HMD200 takes in instruction information from the user 10's voice and can execute actions in response to the instruction information in a user-friendly manner.

[0069] The audio output device 1543 can output sound from its speaker based on audio data, and can inform the user 10 of notification information by voice. For example, if the user 10 is located or facing a different position and direction than the viewing position and direction before the pause within the control area range 206, it will emit a voice message informing the user that their viewing position and direction are different from the viewing position and direction before the pause.

[0070] The vibration generating device 1541 generates vibrations under control from the processor 1520, converting notification instructions transmitted to the user 10 by the HMD 200 into vibrations. The vibration generating device 1541 transmits vibrations to the user's head wearing the HMD 200, making the notification instructions known to the user 10 and improving usability.

[0071] The communication device 1544 is a communication interface that performs wireless communication with other devices via short-range wireless communication. The communication device 1544 includes communication processing circuits and antennas corresponding to various predetermined communication interfaces, and performs transmission and reception of various information, control signals, etc. It may also include a telephone communication network.

[0072] Timer 1545 measures the time until user 10 leaves the control area range 206 without looking at the video virtual object 205, the time of user movement 1303 from the user's current position to the position where they leave the control area range, and the elapsed time after user 10 leaves the control area range 206. The time measured by Timer 1545 is recorded by the processor 1520 in memory 1530.

[0073] The virtual object processing unit 1521 generates a three-dimensional virtual video object 205, such as a soccer match video, and places it in three-dimensional space. It also records information about the position and orientation of the placed virtual video object 205 in the memory 1530. Furthermore, if the user 10 is outside the control area range 206 for a predetermined time, the virtual object processing unit 1521 performs the following processing. That is, while maintaining the relative position and orientation relationship between the virtual video object and the user and the user's line of sight when paused, the virtual object processing unit 1521 changes the position and orientation of the virtual video object in accordance with the movement of the HMD 200. Then, it newly fixes the virtual video object in the local coordinate system of the HMD 200. Furthermore, when fixing the virtual video object 205 in the local coordinate system of the HMD 200, if the size of the virtual video object is larger than a predetermined threshold, the virtual object processing unit 1521 performs the following processing. In other words, the virtual object processing unit 1521 shrinks the video virtual object 205, or extracts and arranges a predetermined area including the point of focus on the recorded video virtual object by the gaze predetermined range processing unit 1524 described later. Furthermore, when user 10 re-enters the control area range 206 after pausing the video virtual object, the virtual object processing unit 1521 performs the following processing. Specifically, it generates and displays virtual objects that show the relative position and direction relationship between the video virtual object 205 before the recorded video virtual object pause and user 10 and user 10's line of sight, as well as the user 10's gaze points 232 on the video virtual object 205. For example, the virtual object processing unit 1521 generates images of virtual objects such as circles, arrows, and points that show the above information and superimposes them onto the 3D space. It also generates images of virtual object messages that show the reasons for selecting playback position candidates for the video virtual object and information about the playback position of the video virtual object, and superimposes them onto the 3D space. In addition, it generates images of virtual objects such as circles, arrows, and points that show information about the position and direction (each viewing position and direction) relationship between the video virtual object and the user at the playback position candidates, and superimposes them onto the 3D space.

[0074] The playback control processing unit 1522 pauses the operation of the virtual video object 205 when the user 10 moves a predetermined distance away from it. When the user 10 approaches within the predetermined distance from the virtual video object 205, the playback control processing unit 1522 performs the following processing. That is, it controls the playback operation of the paused virtual video object 205 based on the recorded relative position information, information on the user's position and gaze after approaching within the predetermined distance, and information on candidate playback positions for the virtual video object 205. Examples of playback operation control include playing it as is or rotating the virtual video object 205 before playback. It also rewinds the virtual video object 205 by a predetermined time and plays it back based on information on the user's viewing importance of the virtual video object and the viewing importance of the playback position of the virtual video object. Furthermore, the playback control processing unit 1522 controls the playback and stop operations of the virtual video object 205 fixedly positioned in the local coordinate system of the HMD 200.

[0075] The mutual positioning information processing unit 1523 acquires the relative position and orientation relationship between the virtual video object and the user's position and the user's line of sight before the virtual video object is stopped, generates mutual positioning information, and records it in the memory 1530. For example, as mutual positioning information, it records the position and orientation of the user 10 or the user 10's line of sight relative to the virtual video object 205 at the location where the user 10 stayed before leaving the control area range 206. Here, the location where the user stayed is, for example, the starting point of movement when the user 10 moved outside the control area range 206. This location where the user stayed may be the position where, based on information from various sensors (for example, positioning sensor 1503, acceleration sensor 1504, gyro sensor 1505), the amount of change in sensor values ​​is smaller than a predetermined value and it is determined that the user 10 has not moved. Also, if the user 10 moves outside the control area range 206 without facing the direction of the virtual video object 205 for a predetermined period of time, the mutual positioning information processing unit 1523 performs the processing described below. In other words, the mutual arrangement information processing unit 1523 records the location where the user 10 stayed before leaving the control area range 206, and the direction from that location toward the video virtual object 205. Here, the direction from the current position toward the virtual video object 205 can be acquired as appropriate. For example, the orientation of the HMD 200 displaying the virtual video object 205 before stopping can be detected by the accelerometer 1504 or gyroscope 1505, and the detected orientation of the HMD 200 can be recorded as the direction from the current position toward the virtual video object 205. The orientation of superimposed virtual objects may also be taken into consideration. In this way, the mutual arrangement information processing unit 1523 records mutual arrangement information using information about the user 10's position and gaze toward the virtual video object 205 that was acquired and recorded before the virtual video object 205 was paused. That is, the mutual arrangement information processing unit 1523 selects and acquires the user 10's viewing position and direction toward the virtual video object 205 before pausing from this information and records it as mutual arrangement information. The mutual arrangement information may also be recorded as position and direction information having a predetermined range.

[0076] The gaze predetermined range processing unit 1524 selects a gaze predetermined range 233 centered on the user's gaze point 232 before stopping, which was identified by the left eye gaze sensor 201 and the right eye gaze sensor 202, generates gaze predetermined range information, and records it in the memory 1530. For example, if the position of the user 10's gaze relative to the virtual image object 205 did not move for a predetermined period of time, the gaze predetermined range processing unit 1524 identifies the area including the immobile gaze position as the gaze point 232. Alternatively, if the amount of movement of the user 10's gaze relative to the virtual image object 205 during a predetermined period of time was less than or equal to a predetermined threshold, the gaze predetermined range processing unit 1524 identifies the area including the position during gaze movement below the predetermined threshold as the gaze point 232.

[0077] The viewing-optimal placement processing unit 1525 selects a suitable viewing position and direction for the virtual video object as viewing-optimal placement information and records the selected viewing-optimal placement information in the memory 1530. For example, the viewing-optimal placement information may include the user's position and point of gaze before pausing. It may also include a suitable viewing position and direction for a virtual object or other scene that the user was fixated on within the video scene of the virtual video object at the time of pausing. Furthermore, it may include a suitable viewing position and direction for a specific virtual object that the user was fixated on within the video scene of the virtual video object at the time of pausing, or a specific virtual object designated in advance by the user. Here, the point of gaze and the virtual object being fixed on can be obtained, for example, based on the left eye gaze sensor 201 and the right eye gaze sensor 202. The direction to a specific virtual object may be estimated by tracking the specific virtual object. The user's viewing position is a position from which the target point of gaze can be viewed optimally, and may be estimated, for example, based on the position of each virtual object.

[0078] The playback position candidate processing unit 1526 selects candidate video playback positions for rewinding and playing back the video virtual object and records them in the memory 1530. Candidate video playback positions may be selected based on information such as the length and frequency of the user's attention, and highlight scenes (including information about other viewers' viewing, such as the scene with the highest total number of views). Alternatively, candidate video playback positions may be selected based on information such as the relationship between the video virtual object and the user's position or the user's gaze (viewing position and direction) at the playback position candidate. Furthermore, in addition to such viewing conditions, candidates may be selected based on the content of the video virtual object (for example, information about highlight tags indicating highlights attached to the video virtual object, or information about the actions of the object that is the focus of attention in this highlight).

[0079] The surrounding conditions processing unit 1527 records the image captured by the camera 203 or displayed on the display processing device 1508 within the control area as placement location surrounding information 1539 in the memory 1530. Then, based on the recorded information, the surrounding conditions processing unit 1527 determines whether or not there are specific viewing locations such as chairs or other viewers around the virtual video object (within the control area). Note that the objects to be determined as specific viewing locations such as chairs or other viewers may be controlled to apply not only to real objects and people, but also to virtual objects other than the virtual video object.

[0080] The placement feasibility processing unit 1528 analyzes the distance measurement results from the distance measuring sensor 1503 and the images captured by the camera 203 or displayed on the display processing device 1508. The placement feasibility processing unit 1528 then identifies the distance from the user 10 to the real object or another virtual object, the size of the real object or another virtual object, the presence or absence of a horizontal surface, and its type (e.g., a desk or a table), and determines whether the virtual video object can be placed.

[0081] With the above configuration, the HMD200 performs the following basic operations. When the user moves a predetermined distance away from the three-dimensional virtual video object 205, which is placed in three-dimensional space by the virtual object processing unit 1521, the playback control processing unit 1522 temporarily pauses the operation of the virtual video object. The mutual positioning information processing unit 1523 also acquires and records the mutual position and direction relationship between the virtual video object and the user or the user's line of sight before the pause as mutual positioning information. After the pause, the user approaches within a predetermined distance from the virtual video object. At this time, the playback control processing unit 1522 controls the playback operation of the virtual video object based on the recorded mutual positioning information, information on the user's position and the direction of the user's line of sight after approaching within the predetermined distance, and information on candidate playback positions for the virtual video object 205. The playback operation control includes playing the virtual video object as is, playing the virtual video object rotated, or playing it back after a predetermined time.

[0082] Furthermore, the HMD200 uses a gaze-determined range processing unit 1524 to select a gaze-determined range centered on the user's gaze point before pausing, and generates and records it as gaze-determined range information. After pausing, if it is detected that the user's gaze point has entered the gaze-determined range indicated by the gaze-determined range information, the playback operation of the virtual video object is resumed.

[0083] Furthermore, in the HMD200, the viewing-optimized placement processing unit 1525 selects a suitable viewing position and direction for the virtual video object as viewing-optimized placement information when the operation of the virtual video object is stopped, and notifies the user of the selected viewing-optimized placement information. The viewing-optimized placement information includes the user's position and gaze point before pausing, and a suitable viewing position and direction for the virtual object that the user was gazing at within the virtual video object's video scene at the time of pausing. It also includes a suitable viewing position and direction for a specific virtual object that the user was gazing at within the virtual video object's video scene at the time of pausing, or a specific virtual object that has been designated in advance.

[0084] Furthermore, the HMD200 uses a playback position candidate processing unit 1526 to record candidate video playback positions for rewinding and playing back the video virtual object, as well as candidate viewing positions and directions at those video playback positions. Candidate playback positions are selected and recorded based on information such as the length and frequency of the user's attention to the video virtual object, highlight scenes (including information about other viewers' viewing, such as the scene with the highest total number of views), and information about each viewing position and direction. In other words, candidate playback positions are selected and recorded based on the viewing status of the video virtual object and the content of the video virtual object (for example, information about highlight tags indicating highlights assigned to the video virtual object, and information about the actions of the object highlighted). The playback operation of the video virtual object is then resumed from the video playback position selected by the user from among the selected and recorded playback position candidates.

[0085] Furthermore, the virtual object processing unit 1521 causes the user to move a predetermined distance away from the three-dimensional virtual video object 205, such as a soccer match video, which is placed in three-dimensional space. Then, the playback control processing unit 1522 temporarily pauses the operation of the virtual video object, and the mutual position information processing unit 1523 acquires and records the mutual position and direction relationship between the virtual video object and the user's position and line of sight before the pause as mutual position information. Subsequently, if the user has been at a predetermined distance from the virtual video object for a predetermined time, the following processing is performed in the HMD 200. That is, the virtual object processing unit 1521 fixes the position and direction of the virtual video object in the local coordinate system of the HMD so that it follows the movement of the HMD, based on the mutual position information recorded by the mutual position information processing unit 1523. Furthermore, the playback control processing unit 1522 controls the playback and stop operation of the virtual video object.

[0086] Furthermore, when the virtual object processing unit 1521 fixes the virtual image object in the local coordinate system of the HMD, if the size of the virtual image object is larger than a predetermined threshold, the HMD 200 performs the following processing: That is, it shrinks the virtual image object, or the gaze predetermined range processing unit 1524 extracts a predetermined range including the gaze point on the recorded virtual image object and fixes it in the local coordinate system of the HMD.

[0087] Furthermore, if the surrounding conditions processing unit 1527 determines that a specific viewing location or other viewers exist around the placed virtual video object, the HMD 200 performs the following processing: that is, it either does not perform the fixed placement of the virtual video object in the HMD's local coordinate system by the virtual object processing unit 1521, or it controls the virtual video object to be duplicated and fixedly placed in the HMD's local coordinate system.

[0088] Furthermore, the user moves a predetermined distance away from the three-dimensional virtual video object 205, such as a soccer match video, which is placed in three-dimensional space by the virtual object processing unit 1521. Then, the playback control processing unit 1522 temporarily pauses the operation of the virtual video object, and the mutual position information processing unit 1523 acquires and records the mutual position and orientation relationship between the virtual video object and the user's position and the user's line of sight before the pause as mutual position information. Subsequently, if the placement feasibility processing unit 1528 determines that the virtual video object can be newly placed when the user approaches another possible placement location, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 controls the virtual video object to be newly fixed in place while maintaining the mutual position and orientation relationship between the virtual video object and the user's position and the user's line of sight at the time of the pause, and then resumes its playback operation.

[0089] Thus, when viewing a three-dimensional virtual image object placed in a specific location, it is possible to realize an information processing device that allows users to easily and conveniently stop and resume viewing at an appropriate timing according to the user's situation and the placement of the three-dimensional virtual image object, without requiring any special operation from the user. For example, even if an urgent matter arises while a user wearing an HMD is viewing, they can stop and resume the operation of the virtual image object at an appropriate time and with ease of use.

[0090] While the above explanation uses HMDs as a specific example of an information processing device, it goes without saying that all devices with similar functions are included, and similar effects and benefits can be obtained with other information processing devices such as smartphones and smartwatches.

[0091] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0092] Furthermore, the numbers and messages appearing in the text and diagrams are merely examples, and using different ones will not impair the effects of the present invention.

[0093] Furthermore, each of the above configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above configurations, functions, etc., may be implemented in software by having the processor interpret and execute programs that realize each function. The processor includes transistors and other circuits and is considered circuitry or processing circuitry. Information such as programs, tables, and files that realize each function may be stored in memory, recording devices such as hard disks and SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs, or in devices on a communication network. Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is safe to assume that almost all components are interconnected.

[0094] The user may freely set the range and shape of the control area. Alternatively, the user may select from several pre-registered patterns as appropriate. The information regarding the range of the control area determined by the user is stored in memory 1530 and used for processing by the HMD200. [Explanation of Symbols]

[0095] 10: User, 200: Head-mounted display (HMD), 201: Left eye gaze sensor, 202: Right eye gaze sensor, 203: Camera, 204: Physical object (desk), 205: Virtual image object, 206: Control area range, 208: Display screen, 215: Positioning sensor, 216: Geomagnetic sensor, 1503: Distance sensor, 1504: Accelerometer, 1505: Gyroscope, 1507: Operation input interface, 1508: Display processing device, 1520: Processor, 1521: Virtual object processing unit, 1522: Playback control processing unit, 1523: Mutual arrangement information processing unit, 1524: Determined gaze range processing unit, 1525: Suitable viewing arrangement processing unit, 1526: Playback position candidate processing unit, 1527: Surrounding conditions processing unit, 1528: Placement feasibility processing unit, 1530: Memory, 1541: Vibration generating device, 1542: Audio input device, 1543: Audio output device, 1544: Communication device, 1545: Timer, 1550: Bus

Claims

1. An information processing device that is attached to a user, A processor that controls the playback of three-dimensional virtual image objects, A positioning sensor that detects the user's location, A gaze sensor that detects the direction of the user's gaze, Equipped with memory, The aforementioned processor, The memory records user information including the placement position where the replayed video virtual object is placed, the user's position detected by the positioning sensor, and the user's gaze direction detected by the gaze sensor. When the positioning sensor detects that the user has moved outside a predetermined distance from the placement location, the system controls the playback of the virtual video object to stop. When the user enters the range after the playback of the virtual video object has been stopped, the system controls the system to resume playback of the virtual video object based on the user information recorded in the memory and the position detected by the positioning sensor or the direction of gaze detected by the gaze sensor. An information processing device characterized by the following:

2. An information processing apparatus according to claim 1, The processor, after stopping the playback of the virtual video object, resumes playback of the virtual video object if the user enters the range and the position detected by the positioning sensor and the gaze direction detected by the gaze sensor correspond to the user information recorded in the memory. An information processing device characterized by the following:

3. An information processing apparatus according to claim 1, The processor controls the system to issue a notification to the user if, after stopping the playback of the virtual video object, the user enters the range and the position detected by the positioning sensor or the direction of gaze detected by the gaze sensor differs from the user information recorded in the memory. An information processing device characterized by the following:

4. An information processing apparatus according to claim 1, The aforementioned processor, If the amount of change in the user's gaze direction detected by the gaze sensor is below a threshold within a predetermined time, the range of the virtual image object in the gaze direction is recorded in the memory as the gaze range. After the playback of the virtual video object is stopped, if the user enters the range and the user's line of sight enters the gaze range, the playback of the virtual video object is resumed. An information processing device characterized by the following:

5. An information processing apparatus according to claim 1, The aforementioned processor, The position and orientation suitable for viewing the aforementioned virtual video object are recorded in the memory as suitable placement information. When the user enters the range after the playback of the video virtual object has been stopped, the preferred arrangement information recorded in the memory is notified to the user. An information processing device characterized by the following:

6. The information processing apparatus described in claim 1 is The system includes a display that shows the virtual video object reproduced by the aforementioned processor, The processor controls the rotation of the virtual video object to display it on the display when, after stopping playback of the virtual video object, the user enters the range and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory. An information processing device characterized by the following:

7. The information processing apparatus described in claim 1 is Equipped with a display, The aforementioned processor, Control the display so that the virtual image object is superimposed on the physical object and displayed on the display. If, after playback of the virtual video object is stopped, the user enters the range and the position detected by the positioning sensor or the gaze direction detected by the gaze sensor differs from the user information recorded in the memory, the system controls the virtual video object to rotate according to the shape or size of the actual object and display it on the display. An information processing device characterized by the following:

8. An information processing apparatus according to claim 1, The processor controls the system so that when the user enters the range after playback of the virtual video object has stopped, it returns to a position a predetermined time before the position where playback of the virtual video object was stopped and then resumes playback of the virtual video object. An information processing device characterized by the following:

9. An information processing apparatus according to claim 1, The aforementioned processor, Multiple playback positions are recorded in the memory as positions for playing back the aforementioned virtual video object. When the user enters the range after the playback of the video virtual object has been stopped, the user is notified of the multiple playback positions. Controls playback to resume from the playback position selected by the user. An information processing device characterized by the following:

10. An information processing apparatus according to claim 1, The aforementioned processor, If the user does not enter the range even after a predetermined time has elapsed since the playback of the video virtual object was stopped, the user is notified whether or not to resume playback of the video virtual object. When the user selects to resume playback, the system controls the placement of the video virtual object that has resumed playback to a position different from the position recorded in the memory. An information processing device characterized by the following:

11. An information processing device worn by a user, which has the function of generating and displaying three-dimensional virtual image objects, It comprises a processor and a memory unit, The aforementioned processor, The mutual positional information, which is the relative positional and directional relationship between the virtual image object and the user's position and line of sight, is recorded in the memory unit. When the user moves away from the virtual video object and the distance between the virtual video object and the user exceeds a predetermined distance, Stop playback of the aforementioned video virtual object, After a predetermined elapsed time from the aforementioned stop, the virtual video objects are positioned in accordance with the movement of the information processing device based on the mutual arrangement information recorded in the storage unit. Controlling the playback operation of the arranged virtual video object, An information processing device characterized by the following:

12. An information processing apparatus according to claim 11, The aforementioned processor, The user is notified whether or not to play the operation of the placed virtual video object. If the user who has received the notification chooses to play the operation of the virtual video object that has been placed, The playback operation of the arranged virtual video object is performed. An information processing device characterized by the following:

13. An information processing apparatus according to claim 11, The aforementioned processor, If it is determined that the size of the aforementioned virtual video object is larger than a predetermined threshold, the size of the virtual video object is reduced and it is positioned accordingly. An information processing device characterized by the following:

14. An information processing apparatus according to claim 11, The aforementioned processor, Before the aforementioned halt, the storage unit records the point of focus that the user was looking at in the virtual video object. When it is determined that the size of the virtual video object is larger than a predetermined threshold, the virtual video object is arranged by extracting the range of a predetermined area, including the point of focus, that has been recorded in the storage unit. An information processing device characterized by the following:

15. A method for processing information of a device worn and used by a user, It plays back three-dimensional virtual objects, The regenerated virtual video object is placed and displayed so as to be superimposed on the physical object. The system records user information including the placement location of the virtual video object, the user's position, and the user's line of sight. When it is detected that the user has moved outside a predetermined distance from the placement location, playback of the video virtual object is stopped. After stopping the playback of the video virtual object, when the user enters the range, playback of the video virtual object is resumed based on the recorded user information. An information processing method characterized by the following:

Images