Information processing apparatus and information processing method

Abstract

An information processing device and an information processing method worn and used by a user reproduce a stereoscopic video virtual target, are arranged so as to overlap with an entity object and display the reproduced video virtual target, and record an arrangement position at which the video virtual target is arranged, and user information including at least one of a position and a line-of-sight direction of the user. When it is detected that the user is out of the range of the predetermined distance from the arrangement position, reproduction of the video virtual target is stopped. When the user enters the range after the playback of the video virtual target is stopped, the playback of the video virtual target is restarted on the basis of the recorded user information.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method. Background Art

[0002] In recent years, augmented reality (AR) technology, which adds digital information to the real world and reflects virtual objects (virtual targets) in a virtual space created using CG (Computer Graphics) and the like into the real space and expands it, has been widely used. As a device using this technology, information processing devices that can easily handle virtual targets while three-dimensionally identifying the real space have become popular. In addition, as an example, a head mounted display (HMD) that is worn on the head and has a display unit, a camera unit, etc. can be cited.

[0003] As a method for pausing and reproducing the action of an image of a virtual object (hereinafter sometimes referred to as an image virtual object), the following is described in Patent Document 1. That is, Patent Document 1 describes that "the reproduction unit 107 reproduces the animation data stored in the storage unit 104 (301). The composite reality processing unit 103 obtains the user's viewpoint position (or the user's position) and the direction of his / her line of sight (S302) by using the position and posture estimation result of the self-position and posture estimation unit 106 and the line of sight tracking function in parallel with the reproduction processing by the reproduction unit 107. The composite reality processing unit 103 calculates the offset Φ of the line of sight from the animation reproduction screen G and the distance D between the user's own position and the animation reproduction screen G. The reproduction unit 107 adjusts the reproduction speed according to the offset Φ and the distance D (S303). For example, when the offset Φ is a predetermined angle offset and the distance D is greater than a predetermined value, the reproduction unit 107 reduces the reproduction speed or stops the reproduction. In addition, the conditions for the offset Φ and the distance D may be at least one of them. This is a reference for determining whether the user is viewing the animation reproduction screen G."

[0004] In addition, Patent Document 2 states that "The user terminal 1500 has a function of obtaining measurement data obtained by measuring the position and posture of the user terminal 1500, is configured to a subject target, a virtual camera, and a virtual three-dimensional space, and controls the position and posture of the virtual camera in conjunction with the measurement data. Furthermore, the user terminal 1500 generates a virtual space image captured by the virtual camera and displays it as a monitoring image, while recording camera working data 550 that can reproduce the position and posture of the virtual camera.", "The reproduced image generation control unit 232 controls the generation of an image of the virtual three-dimensional space from the virtual camera at a given reproduction timing of the camera operation, i.e., a reproduced image, based on the camera working data 550 recorded by the recording control unit 224.".

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent document 1: International Publication No. WO2021 / 132168A1

[0008] Patent Document 2: Japanese Patent Application Publication No. 2020-119334 Summary of the invention

[0009] As described above, Patent Document 1 shows that the stop and playback control of the image virtual object screen is performed according to the offset and distance of the user's line of sight relative to the position / direction of the arranged plane image virtual object. On the other hand, it does not consider the case of a three-dimensional image virtual object that can be viewed from multiple positions and directions. Therefore, it is considered to stop viewing or restart viewing of the three-dimensional image virtual object in an appropriate state and at a certain time.

[0010] As described above, Patent Document 2 shows that the reproduction of an image virtual target is controlled based on information related to the camera operation (camera position / posture). On the other hand, the user's line of sight with respect to the image virtual target, the distance from the user, etc., are not disclosed at all. Therefore, it is considered that it is impossible to control the stop / reproduction of the image virtual target based on the offset / distance of the user's line of sight relative to the position / direction of the image virtual target. In addition, in Patent Document 2, although the technology related to appropriate "animation production" is shown, the technology related to appropriate "audio-visual" is not shown, and there is a problem that it is difficult to stop or restart "audio-visual" at an appropriate timing.

[0011] Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object thereof is to provide an information processing device and an information processing method that can easily and conveniently stop or restart viewing of a virtual object in a stereoscopic image.

[0012] Among the inventions disclosed in this application, the outline of representative inventions will be briefly described as follows.

[0013] (1) An information processing device, worn by a user, comprising: a processor, which controls the reproduction of a three-dimensional image virtual object; a positioning sensor, which detects the position of the user; a line of sight sensor, which detects the line of sight of the user; and a memory. The processor records the configuration position of the image virtual object to be reproduced, and user information including at least one of the position of the user detected by the positioning sensor and the line of sight of the user detected by the line of sight sensor, in the memory. In addition, when the positioning sensor detects that the user has left a range of a predetermined distance from the configuration position, the processor controls the reproduction of the image virtual object to be stopped, and when the user enters the above range after the reproduction of the image virtual object is stopped, the processor controls the reproduction of the image virtual object to be restarted based on the user information recorded in the memory and the position detected by the positioning sensor or the line of sight detected by the line of sight sensor.

[0014] (2) In the information processing device described in (1), when the user enters the above-mentioned range after the processor stops the reproduction of the image virtual target, the reproduction of the image virtual target is started again when the position detected by the positioning sensor and the line of sight direction detected by the line of sight sensor correspond to the user information recorded in the memory.

[0015] (3) In the information processing device described in (1), when the user enters the above-mentioned range after the processor stops reproducing the image virtual target, if the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory, the processor controls in a manner to notify the user.

[0016] (4) In the information processing device described in (1), when the change in the user's line of sight direction detected by the line of sight sensor is below a threshold during a predetermined time period, the processor records the range of the image virtual target in the line of sight direction as a gaze range in the memory, and when the user enters the above range and the user's line of sight enters the gaze range after the reproduction of the image virtual target is stopped, the reproduction of the image virtual target is started again.

[0017] (5) In the information processing device described in (1), the processor records the position and direction suitable for the audio-visual image virtual target as appropriate configuration information in the memory, and when the user enters the above-mentioned range after stopping the reproduction of the image virtual target, the appropriate configuration information recorded in the memory is notified to the user.

[0018] (6) In the information processing device described in (1), there is a display for displaying a virtual image target reproduced by a processor, and after the reproduction of the virtual image target is stopped, when the user enters the above-mentioned range and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory, the processor controls the virtual image target to rotate and display it on the display.

[0019] (7) In the information processing device described in (1), a display is provided, and the processor controls the image virtual target to be overlapped with the real object and displayed on the display. After the reproduction of the image virtual target is stopped, when the user enters the above-mentioned range and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory, the image virtual target is controlled to be rotated according to the shape or size of the real object and displayed on the display.

[0020] (8) In the information processing device described in (1), when the user enters the above-mentioned range after stopping the reproduction of the image virtual target, the processor controls the image virtual target to return to a position a predetermined time earlier than the position where the reproduction of the image virtual target was stopped and restart the reproduction of the image virtual target.

[0021] (9) In the information processing device described in (1), the processor records multiple reproduction positions as positions for reproducing the image virtual target in the memory, and when the user enters the above-mentioned range after stopping the reproduction of the image virtual target, the multiple reproduction positions are notified to the user, and the reproduction of the image virtual target is controlled to be restarted from the reproduction position selected by the user.

[0022] (10) In the information processing device described in (1), when the user does not enter the range even after a predetermined time has passed after stopping the reproduction of the image virtual target, the processor notifies the user whether to restart the reproduction of the image virtual target. When the user chooses to restart the reproduction, the processor controls the image virtual target to be reproduced again in a manner such that the image virtual target is configured at a position different from the configuration position recorded in the memory.

[0023] (11) An information processing device having the function of generating and displaying a three-dimensional image virtual target, which is worn by a user, wherein the information processing device includes a processor and a storage unit, the processor records the mutual configuration information of the position of the image virtual target and the user and the mutual positional direction relationship of the line of sight in the storage unit, and when the user leaves the image virtual target and the distance between the image virtual target and the user becomes greater than a predetermined distance, the reproduction of the image virtual target is stopped, and after a predetermined time has passed since the stop, the image virtual target is configured in a manner that follows the activity of the information processing device based on the mutual configuration information recorded in the storage unit, and the reproduction action of the configured image virtual target is controlled.

[0024] (12) In the information processing device described in (11), the processor notifies the user whether to reproduce the action of the configured image virtual target, and when the user who has received the notification chooses to reproduce the action of the configured image virtual target, the reproduction action of the configured image virtual target is executed.

[0025] (13) In the information processing device described in (11), when the processor recognizes that the size of the image virtual object is larger than a predetermined threshold, the processor reduces the size of the image virtual object and arranges it.

[0026] (14) In the information processing device described in (11), the processor records the user's gaze point on the image virtual target in a storage unit before stopping, and when it is recognized that the size of the image virtual target is larger than a predetermined threshold, configures the image virtual target to extract the range of the predetermined area including the gaze point recorded in the storage unit.

[0027] (15) An information processing method for a device worn and used by a user, wherein a three-dimensional image virtual target is reproduced, the reproduced image virtual target is arranged and displayed in a manner overlapping with a physical object, a configuration position of the configured image virtual target and user information including at least one of the user's position and the user's line of sight direction are recorded, and when it is detected that the user leaves a range of a predetermined distance from the configuration position, the reproduction of the image virtual target is stopped. When the user enters the above range after the reproduction of the image virtual target is stopped, the reproduction of the image virtual target is restarted based on the recorded user information.

[0028] By using the technology of the present invention, it is possible to realize an information processing device and an information processing method that can easily and conveniently stop or restart viewing of a virtual object in a stereoscopic image.

[0029] In addition, other problems, structures and effects than those described above will become more apparent through the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an example of a diagram schematically illustrating an example of an information processing device according to the present embodiment.

[0031] Figure 2 The appearance schematically illustrates the Figure 1 An example of a diagram of a specific display screen of the embodiment described in .

[0032] Figure 3 Yes Description Figure 1 An example of a diagram showing a control operation of the information processing device according to the present embodiment is shown.

[0033] Figure 4(a) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0034] FIG. 4( b ) is a diagram schematically illustrating an example of a display screen during a control operation of the information processing device according to the present embodiment shown in FIG. 4( a ).

[0035] Figure 5(a) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0036] Figure 5(b) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0037] Figure 6 This is an example of a flowchart for explaining the basic operation of the information processing device according to the present embodiment.

[0038] Figure 7 Yes Description Figure 1 An example of a diagram showing a recording operation of the information processing device according to the present embodiment is shown.

[0039] Figure 8 Yes Description Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0040] Fig. 9 Yes Description Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0041] Figure 10(a) is a diagram illustrating Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0042] FIG. 10( b ) is an example of a diagram for explaining a result of a control operation of the information processing device according to the present embodiment shown in FIG. 10( a ).

[0043] Fig.11 Yes Description Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0044] Fig.12 Yes Description Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0045] Fig.13 Yes Description Figure 1Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0046] Fig.14 Yes Description Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0047] Figure 15(a) is a diagram illustrating Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0048] Figure 15(b) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0049] Figure 15(c) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0050] Figure 15(d) shows Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0051] Figure 15(e) is a diagram illustrating Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0052] Figure 15(f) is a diagram illustrating Figure 1 Another example of a diagram showing the control operation of the information processing device according to the present embodiment is shown.

[0053] Fig.16 This is a block diagram showing a configuration example of an HMD as an example of the information processing device according to the present embodiment. DETAILED DESCRIPTION

[0054] An example of an embodiment of the present invention is described below using the accompanying drawings. As a specific example of the information processing device involved in this embodiment, a head mounted display (HMD) worn on the user's head and displaying information in real space and virtual space for visual recognition is described as an example. Figure 1 , Figure 3 , Figure 4(a), Figure 5(a), Figure 5(b), Figure 7 FIG. 1 is an example of a diagram schematically showing the appearance of an information processing device according to the present embodiment. Figure 2 4(b) is an example of a diagram for explaining a display screen within the field of view of the information processing device according to the present embodiment. Figure 1In the embodiment, HMD 200 is worn on the head of user 10. HMD 200 includes left eye sight sensor 201 and right eye sight sensor 202 for detecting sight lines of left and right eyes of user 10, camera 203 for photographing the outside world, positioning sensor 215 for detecting the position on the earth, and geomagnetic sensor 216 for detecting the direction of HMD orientation.

[0055] Compared to an optical see-through type HMD in which the user 10 directly uses his eyes to visually recognize physical objects in the field of view around the front, the image see-through type HMD uses the camera 203 to capture physical objects in the field of view around the front, and displays the captured images of the physical objects on the display. In addition to the physical objects, the HMD 200 also generates three-dimensional image virtual objects, which are three-dimensionally arranged and displayed within the field of view of the user 10. As a result, the image virtual objects appear to the user 10 as virtual objects that exist in the real world.

[0056] In addition, the HMD 200 records information on the position and direction of the arranged three-dimensional image virtual object. Figure 1 The image virtual object 205, which is an image of a soccer game, is arranged in the space on the table 204, which is a physical object. Although not shown in the figure, it is preferable to display a selection screen for the user 10 to select a physical object to arrange the image virtual object so that the user 10 can arrange it at any place. In addition, the HMD 200 detects the position of the user 10 using the positioning sensor 215 provided therein, and detects the direction in which the user 10 is facing using the geomagnetic sensor 216 provided therein.

[0057] In user 10 Figure 1 As shown in FIG. 2 , when the user 10 is at a position 207 inside the control section range 206 within a predetermined distance from the image virtual target 205, the user 10 can visually recognize the image virtual target 205. Figure 2 As shown, the user 10 can visually recognize the image of a football game as an example of the image virtual object 205 arranged on the table 204 in the display screen 208 within the field of view. In addition, the table 204 can also be a virtual object independent of the image virtual object 205. In this case, that is, when the image virtual object 205 is displayed, the HMD 200 can grasp the position / direction of the image virtual object 205 arranged in the three-dimensional space. In addition, the HMD 200 can detect the position / direction of the user 10 using the positioning sensor 215 and the geomagnetic sensor 216, and detect the line of sight of the user 10 using the left eye line of sight sensor 201 and the right eye line of sight sensor 202. Therefore, the HMD 200 can determine and obtain the relationship between the position and direction of the image virtual object 205 and the user 10, and the relationship between the direction of displaying the image virtual object 205 and the line of sight direction of the user 10.

[0058] Here, user 10 Figure 3 As shown by the arrow 210 in FIG. 2 , the image virtual object 205 moves away from the predetermined distance and moves to the position 209 outside the control section range 206. At this time, the HMD 200 pauses the action of the image virtual object 205, which is the soccer image in the game. Thus, the action of the image virtual object can be immediately paused without the user performing a special operation, simply by the user 10 moving outside the control section range 206. In addition, the HMD 200 determines the relationship between the position and direction of the image virtual object 205 and the user 10, and the relationship between the direction in which the image virtual object 205 is displayed and the direction of the user 10's line of sight, and records the mutual position / direction relationship before the pause as mutual configuration information.

[0059] Here, the HMD 200 records the information described below as the mutual configuration information. For example, the HMD 200 records the relative position / direction relationship of the user 10 at the stay position (the start position of the movement when the user 10 moves to the outside of the control segment range 206) where the user 10 stays before leaving the control segment range 206 with respect to the image virtual target 205. In addition, in the case where the user 10 moves to the outside of the control segment range 206 without moving in the direction of the image virtual target 205 during a predetermined period of time, the HMD 200 records the information described below as the mutual configuration information. For example, the HMD 200 records the stay position where the user 10 stays before leaving the control segment range 206 and the direction from the stay position to the image virtual target 205.

[0060] In this way, HMD200 extracts and obtains the audiovisual position / direction of the user 10 relative to the image virtual target 205 before the pause from the information related to the position of the user 10 relative to the image virtual target 205 or the line of sight direction of the user 10 recorded before the pause. Moreover, HMD200 records the audiovisual position / direction of the user 10 relative to the image virtual target 205 before the pause as the mutual configuration information. In addition, as an example, the stay position can be set to the position where the user 10 stays when viewing the image virtual target, that is, the position where the user 10 continues to view the image virtual target 205 without moving for a predetermined time. In addition, the mutual configuration information can also be recorded as information on the position / direction having a predetermined range. Thus, it is possible to record information indicating a rough position and direction.

[0061] Alternatively or additionally, the arrangement position where the video virtual object 205 is arranged and the position and sight line direction of the user as user information may be recorded separately.

[0062] After the action of the image virtual object 205 is paused, the user 10 Figure 3The HMD 200 approaches the image virtual object 205 as shown by the arrow 211. At this time, when the user 10 comes within the predetermined distance from the image virtual object 205 and is located and facing the viewing position / direction before the pause, the HMD 200 restarts the reproduction action of the image virtual object 205 of the football game. That is, when the user 10 is within the predetermined range based on the recorded mutual arrangement information and faces the same direction as the recorded mutual arrangement information, the HMD 200 restarts the reproduction action of the image virtual object 205. That is, the HMD 200 determines whether the user 10 is facing the same position and direction as before the pause based on the mutual arrangement information recorded in the state before the pause, the position of the user 10 after coming within the predetermined distance, and the information related to the line of sight of the user 10.

[0063] Thus, the reproduction of the image virtual object 205 can be immediately restarted without the user 10 performing a special operation. Furthermore, the user 10 can restart the reproduction of the image virtual object 205 and view it in a state where the approximate viewing state before the user 10 stopped the image virtual object 205 is reproduced. Thus, the HMD 200 can reduce the difference in the visual recognition image of the user 10 before and after the stop caused by the difference in the viewing direction of the user 10 for the image virtual object 205 that is viewed differently according to the angle. Furthermore, the user 10 can continuously view the image virtual object 205.

[0064] On the other hand, after the motion of the video virtual object 205 is paused, the user 10 approaches the video virtual object 205 from a position and direction different from the viewing position / direction of the video virtual object 205 before the pause, as shown by the arrow 221 in FIG. 4(a). When reaching the position 222, the HMD 200 generates a message 223 indicating that the viewing position / direction of the user 10 is different from the viewing position / direction of the user 10 before the pause, and displays it on the display screen 208 within the field of view, as shown in FIG. 4(b), and notifies the user 10. Here, the HMD 200 can determine whether the viewing position / direction is different from that before the pause based on the mutual arrangement information recorded in the state before the pause.

[0065] FIG4(b) is a diagram showing a display screen 208 within the field of view of the HMD 200, and shows a case where a message such as “Notification message viewing position / direction is different…” is generated and displayed as an example of a message 223 displayed in the display screen 208. Thus, the HMD 200 can notify the user 10 that the position / direction of the user 10 reentering the control segment range 206 is different from the viewing position / direction of the image virtual object 205 before the pause. In addition, the HMD 200 can notify the user 10 that a difference occurs in the visual recognition image of the user before and after the pause due to the difference in the viewing direction for the image virtual object 205 that is viewed differently according to the angle.

[0066] In addition, the notification method is not particularly limited, and may be a message as shown in FIG4(b), or may be a display of a symbol indicating the meaning, a warning using sound or vibration, etc. For example, the HMD 200 may make the line indicating the control segment range 206 a thick line or a solid line, or change the color of the displayed line, or make the line flicker for a predetermined time to make a notification.

[0067] The user 10 who receives such a notification can recognize that the position is different from the viewing position before the pause and the direction is different from the viewing direction before the pause. Moreover, the user 10 can choose to restart the playback while keeping the movement of the image virtual object 205 as it is, or restart the playback by controlling the rotation of the image virtual object 205, etc., based on the recognition of this meaning. Moreover, when the user 10 chooses to restart the playback while keeping the movement of the image virtual object 205 as it is, the effect of being able to restart the viewing of the image virtual object 205 from the position / direction that the user 10 wants to view can be obtained.

[0068] When the user 10 moves away from the image virtual object 205 by a predetermined distance and out of the control section range 206 as shown by the arrow 231 in FIG. 5( a ), and the HMD 200 stops the movement of the image virtual object 205, the information related to the line of sight of the user 10 is recorded. That is, as shown in FIG. 5( a ), the HMD 200 extracts a predetermined gaze range 233 centered on the gaze part 232 of the user 10 before stopping determined by the left eye gaze sensor 201 and the right eye gaze sensor 202, and records it as the predetermined gaze range information. In addition, when the line of sight of the user 10 relative to the image virtual object 205 does not move during a predetermined time, the HMD 200 determines the part including the position of the front end of the line of sight as the gaze part 232. Alternatively, when the movement amount of the line of sight of the user 10 relative to the image virtual object 205 is less than a predetermined threshold, the HMD 200 determines the part including the position of the front end of the line of sight as the gaze part 232.

[0069] After the pause, the user 10 moves into the control segment range 206 as shown by the arrow 234 in FIG. 5( b ). Furthermore, the HMD 200 recognizes the position and orientation corresponding to the recorded mutual arrangement information, and when recognizing that the front end 235 of the user's line of sight of the user 10 enters the predetermined gaze range 233 indicated by the predetermined gaze range information, the HMD 200 restarts the reproduction of the image virtual object 205. Thus, the HMD 200 can restart the reproduction of the image virtual object 205 in a state near the gaze portion 232 that the user 10 gazed at when the action of the front end 235 of the user's line of sight toward the image virtual object 205 was stopped. Therefore, after the reproduction is restarted, the user 10 can also gaze at the portion that is substantially the same as before the pause from the viewing position / direction before the pause, and can smoothly view and listen to the image virtual object 205 that is being reproduced again without discomfort from before the pause. Further, the HMD 200 can restart the viewing and listening of the image virtual object 205 in a state in which the state in which the user 10 paid special attention to viewing and listening to the image virtual object 205 before the pause is reproduced.

[0070] As shown in Fig. 5(a) and Fig. 5(b), the video virtual object 205 is a soccer match, and the user 10 watches and listens to the soccer ball. In this case, even when the playback is restarted from a pause, the user 10 can continue to watch and listen to the soccer ball without interruption.

[0071] In addition, when the user 10 re-enters the control section range 206 after pausing the image virtual target 205, the HMD 200 may also perform the display described below. That is, the HMD 200 may also be controlled in a manner to display the pre-recorded image virtual target 205 before the pause and the position of the re-entering user 10, and the mutual position / direction relationship of the user 10's line of sight. Here, there is no particular limitation on the display style of the mutual position / direction relationship as long as the respective relationships can be grasped. For example, the HMD 200 may also display the difference between the position of the virtual target 205 before the pause and the current position of the HMD 200 using a numerical value. Here, when the difference is less than a predetermined amount, the HMD 200 may also notify the meaning through text information display, sound output, etc. For example, the HMD 200 may also display an orientation mark indicating the direction of the image virtual target 205 before the pause, and an orientation mark indicating the current orientation of the HMD 200. Alternatively, the HMD 200 may generate an image of a virtual object such as a circle, an arrow, or a dot indicating the gaze point 232 of the user 10 with respect to the video virtual object 205 , and control the image so as to be displayed in an overlapping manner in a three-dimensional space.

[0072] Thus, when the user 10 re-enters the control section range 206, the user 10 can immediately grasp the positions of the image virtual object 205 and the user 10 before the pause, and the mutual position / direction relationship of the line of sight of the user 10. In addition, the user 10 can immediately grasp the user's gaze position 232 with respect to the image virtual object 205. Thus, when the stereoscopic image virtual object 205 arranged in a specific place can be obtained, the user 10 can stop the viewing and restart the reproduction easily and conveniently by appropriately timing actions corresponding to the situation without special operations.

[0073] Next, basic operations of the information processing device according to the present embodiment will be described using a flowchart. Figure 6 This is an example of a flowchart for explaining the basic operation of the information processing device involved in this embodiment. The basic operation of the information processing device (HMD200 in this example) is appropriately performed using hardware resources described later. Figure 6 In the embodiment of the present invention, the HMD 200 worn by the user 10 reproduces and displays the image virtual target 205 (S701). Furthermore, when the user 10 moves away from the image virtual target 205 by a predetermined distance and moves outside the control segment range 206 (S702), the HMD 200 pauses the reproduction of the image virtual target 205 (S703). At this time, the HMD 200 records the positions of the image virtual target 205 and the user 10 before the stop, and the mutual position / direction relationship of the user 10's line of sight as mutual configuration information. In addition, the HMD 200 records the predetermined gaze range centered on the gaze part before the stop as predetermined gaze range information (S704). Thereafter, in the state where the movement of the image virtual target 205 is paused, the user 10 approaches the image virtual target 205 and enters the inner side of the control segment range 206 within the predetermined distance (S705).

[0074] Here, HMD200 recognizes whether the user 10 is located at the viewing and listening position before the stop and the user 10 is facing the direction before the stop (that is, whether the user 10 is located within the predetermined range based on the recorded mutual arrangement information and the user 10 is facing the predetermined direction). Furthermore, HMD200 recognizes whether the front end of the user 10's line of sight is facing the inside of the predetermined gaze range (S706). When HMD200 recognizes that the user 10 is located at the viewing and listening position before the stop and is facing the direction before the stop, and further, the front end of the user 10's line of sight is facing the inside of the predetermined gaze range, HMD200 restarts the reproduction of the image virtual object 205 (S707). Furthermore, when the display / reproduction of the image virtual object 205 is not completed, HMD200 returns to sequence S702 and repeats this process. When the display / reproduction of the image virtual object 205 is completed, HMD200 ends this process (S708).

[0075] On the other hand, when the user 10 is not at the viewing position before the stop and is not facing the direction before the stop, the HMD 200 notifies the user 10 that the position / direction is different (S709). Here, when it is determined that the user 10 who has received the notification has restarted the reproduction of the image virtual object 205 (S710), the user 10 inputs his / her intention to the HMD 200 by an appropriate method, and the HMD 200 restarts the reproduction of the image virtual object 205 (S707). In addition, as for the method of input by the user 10, for example, a method using button operation, gesture, etc. can be cited. At this time, the HMD 200 can either restart the reproduction of the image virtual object 205 while keeping it as it is, or restart the reproduction by rotating the image virtual object 205, etc., and controlling it to a posture state suitable for viewing and listening.

[0076] When the user 10 determines not to restart the reproduction of the video virtual object 205 (S710), the HMD 200 returns to the sequence S702 and repeats the present process before the display / reproduction of the video virtual object 205 ends. When the display / reproduction of the video virtual object 205 ends, the HMD 200 ends the present process (S708). Through the above-mentioned operations, an information processing device (HMD 200 in this example) is provided that can stop and restart the viewing and listening of the video virtual object in a simple and usable manner without requiring the user to perform special operations.

[0077] Next, use Figure 7 , Figure 8 , Fig. 9 , describing an information processing device for notifying a user of a user's position and gaze point before pausing and an appropriate viewing and listening position / direction of a video virtual object when stopping. Figure 7 , Figure 8 , Fig. 9 in Figure 1 , Figure 2 , Figure 3 4(a), 4(b), 5(a), and 5(b) and the parts shown and denoted by the same reference numerals have the same Figure 1 , Figure 2 , Figure 3 4(a), 4(b), 5(a), and 5(b) are the same actions as those already described, so their detailed descriptions are omitted. Figure 7In the example, the HMD 200 can capture the position of the user 10 using the positioning sensor 215, and can capture the gaze point on the image virtual object 205 that the user 10 is gazing at using the left eye gaze sensor 201 and the right eye gaze sensor 202. Therefore, the HMD 200 notifies the user of the position and gaze point of the user 10 that will be added before the pause as appropriate audio-visual configuration information, so that the user 10 can understand the audio-visual position of the user 10 before the image virtual object 205 stops.

[0078] Next, explain Figure 8 , Fig. 9 In addition, Figure 7 The parts shown in FIG. and assigned the same symbol have the same Figure 7 The same actions as those already described in , so some of their detailed descriptions are omitted. Figure 8 In the example, when the user 10 moves out of the control section range 206 as indicated by the arrow 210, the HMD 200 pauses the action of the image virtual target 205, which is the soccer match image. At this time, the HMD 200 notifies the user 10 to select the position / direction of the appropriate highlight of the image scene in the paused image virtual target 205 as the appropriate audio-visual configuration information. Here, the position / direction of the appropriate highlight of the image scene is the position / direction that can be seen in front of the user without blocking the view of the virtual object or the like that the user is looking at in the image scene. Figure 8 In the example of a soccer match video represented by the video virtual object 205, it is assumed that the user is looking at the soccer ball 801. Therefore, the HMD 200 selects the appropriate viewing position / direction 802 in which the soccer ball being looked at can be viewed as if it were in the hand without being blocked by other virtual objects as the appropriate viewing configuration information, and notifies the user 10 of the information.

[0079] Thus, the HMD 200 can suggest to the user 10 an appropriate viewing position / direction for a certain virtual object in the image virtual target 205 that the user 10 has gazed at. Furthermore, the user 10 can re-enter the image virtual target 205 at the suggested viewing position / direction and start reproducing the image virtual target 205 at an appropriate highlight. For another example, the HMD 200 suggests to the user an appropriate viewing position / direction where a certain football player in the image virtual target 205 that the user has gazed at can be viewed from the front or obliquely instead of from behind, and can be viewed without being blocked by other virtual objects. Therefore, the reproduction can be started again at an appropriate highlight.

[0080] exist Fig. 9In the example, when the user 10 moves outside the control section range 206 as indicated by the arrow 210, the HMD 200 also pauses the action of the video virtual target 205, which is the video of the soccer match. At this time, the HMD 200 selects the appropriate audio-visual position / direction of a specific virtual object in the video of a certain video scene for the video virtual target 205 in the paused state as the appropriate audio-visual configuration information, and notifies the user of the information. Fig. 9 In the example of a soccer match video represented by the video virtual target 205, a soccer player 901 is selected as a specific virtual object. The HMD 200 extracts the soccer player 901 surrounded by four borders 902 as a specific virtual object. Furthermore, the HMD 200 selects an appropriate viewing position / direction 903 as appropriate viewing configuration information so that the movement of the soccer player 901 as a specific virtual object can be viewed as if it were in front of one's eyes and in one's hand without being blocked by other virtual objects, and notifies the user 10 of the information.

[0081] Thus, in the video virtual target 205 viewed by the user 10, the HMD 200 can suggest to the user 10 a position / direction where the user 10 can appropriately view virtual objects, important characters, and other virtual objects that were previously designated by the user 10, regardless of the user 10's gaze point. Furthermore, the user 10 can re-enter the video virtual target 205 at the suggested viewing position / direction and start reproducing the video virtual target 205 at an appropriate highlight. In addition, when selecting the viewing position / direction where the above-mentioned specific virtual objects and specific virtual objects can be appropriately viewed, the HMD 200 can also make a judgment as described below. That is, the HMD 200 can also record in advance the highlighted scenes that are frequently viewed by the viewer and the specific virtual objects that the viewer frequently looks at, for the soccer image as the video virtual target 205, and make a judgment based on the record.

[0082] In addition, as another example, the user 10 may designate a specific virtual object in the viewing of the video virtual object 205, and the HMD 200 may select an appropriate viewing position / direction for the designated specific virtual object as appropriate viewing configuration information and notify the instructing user 10. Thus, the HMD 200 may notify the instructing user 10 of an appropriate viewing position / direction for the specific virtual object arbitrarily selected by the user 10.

[0083] Next, using FIG. 10(a), FIG. 10(b), Fig.11 , Fig.12 , describing an information processing device for performing posture control such as rotating a video virtual object when replaying the video virtual object after stopping. Fig.11 , Fig.12 in Figure 1-Figure 9The parts shown in FIG. and assigned the same symbol have the same Figure 1-Figure 9 10( a) and 10( b) are the same actions as those already described in the control section, so some of their detailed descriptions are omitted. In FIG. 10( a), when the user 10 moves out of the control section range 206 as shown by the arrow 210, the HMD 200 pauses the action of the image virtual target 205, which is the soccer match image. In addition, the HMD 200 records the positions of the image virtual target 205 and the user 10 and the mutual position / direction relationship of the line of sight of the user 10 before the image virtual target 205 is stopped as mutual configuration information. After the pause, the user 10 approaches from the direction 1001 to within a predetermined distance from the image virtual target 205. Here, the HMD 200 rotates the image virtual target 205 according to the direction 1001 that the user 10 approaches and the position / direction relationship of the line of sight of the user 10, which are recorded when the user 10 stopped, according to the position 1001 that the user 10 approaches and the position 1002 after the approach. That is, as shown in FIG. 10( a), the user 10 approaches the soccer match image, which is the image virtual target 205, from the position 1002 on the opposite side in the direction 1001 that is substantially the same as the direction 210 that the user 10 left. In this case, the HMD 200 rotates the soccer game image approximately 180 degrees as indicated by arrows 1003 and 1004 .

[0084] Thus, as shown in FIG. 10( b ), the approaching user 10 can view and listen to the football match video in the same configuration state as before the stop. That is, before the stop and after the restart of the reproduction, the black jersey team players can be seen on the left side of the field of view, and the white jersey team players can be seen on the right side of the field of view, and even after the transition from the stop to the restart of the reproduction, the user 10 can view and listen smoothly without discomfort. The user 10 approaches the image virtual target 205 from a direction different from the viewing direction of the user 10 before the image virtual target 205 stops, or after the user 10 approaches, the user 10 is in a direction different from the viewing direction of the user 10 before the image virtual target 205 stops. Even in such a case, even if the user 10 does not move to the viewing position / direction before the stop, the user 10 can restart the viewing and listening of the image virtual target 205 in a state that reproduces the approximate viewing and listening state taken before the image virtual target 205 stops.

[0085] Fig.11 The figure shows a case where the user 10 approaches the soccer match video as the video virtual target 205 from a direction 1101 that is approximately perpendicular to the direction 210 in which the user 10 left after the pause. Here, it is considered that the video virtual target 205 is rotated about 90 degrees in the directions indicated by arrows 1102 and 1103 according to the direction 1101 in which the user 10 approaches, based on the position / direction relationship of the user 10 and the line of sight of the user 10 when the pause was recorded. In this case, as shown in FIG. Fig.11As shown, the soccer match video 1104 rotated about 90 degrees is exposed from above the table 204. Therefore, when the video virtual object 205 is rotated so that it is not contained by the size and shape of the arrangement place such as the table 204, the HMD 200 does not rotate the video virtual object 205. This can prevent the video virtual object 205 and the arrangement place from being offset due to the rotation of the video virtual object 205.

[0086] Fig.12 The situation in which the user 10 approaches the image virtual object 205 from the direction 1001 after the pause is shown in the same manner as shown in FIG. 10( a). Here, when there is a specific viewing location such as a chair 1201 inside the control section range 206 that includes the image virtual object 205, the HMD 200 does not rotate the football match image that is the image virtual object 205. In this case, since it is highly likely that the user 10 will start viewing the image virtual object 205 again from a specific viewing location while sitting on the chair, the HMD 200 does not intentionally rotate the image virtual object 205, so that the user can view the image virtual object 205 without causing any particular discomfort. In addition, in the case of other viewers, it is beneficial not to rotate the image virtual object 205 in order not to affect the viewing of others. Thus, it is possible to obtain an effect that the reproduction control of the image virtual object 205 and the notification instruction to the user 10 can be performed in consideration of the status of the viewing location of the image virtual object 205.

[0087] Next, use Figure 7 , Fig.13 , describes an information processing device for reversing the reproduction position (reproduction time position) of a virtual image target and reproducing it when the virtual image target is reproduced. When the user approaches the predetermined distance from the paused virtual image target, the information processing device performs processing based on the recorded mutual configuration information, the information related to the reproduction position of the virtual image target 205, the position of the user after approaching the predetermined distance, and the information related to the direction of the user's line of sight. In addition, Fig.13 in Figure 1 -Figure 5, Figure 7-Figure 12 The parts shown in the figure and assigned the same symbols have the same actions as those already described in the figure, so some of their detailed descriptions are omitted.

[0088] When the user 10 is within the control zone range 206, Figure 7As shown, the front end of the line of sight of the user 10 is directed toward the image virtual target 205. Afterwards, when the user 10 intends to move out of the control section range 206, the front end of the line of sight of the user 10 moves away from the image virtual target 205. Here, the HMD 200 records the reproduction position of the image virtual target 205 when the user 10 moves away from the control section range 206. Furthermore, when the user 10 moves out of the control section range 206, the movement of the image virtual target 205 stops. Therefore, when the user 10 approaches the paused image virtual target 205, the HMD 200 performs the processing described below based on the recorded mutual configuration information, the information related to the reproduction position of the image virtual target 205, the position of the user after approaching within a predetermined distance, and the information related to the direction of the user's line of sight. That is, when the HMD 200 restarts the reproduction of the movement of the image virtual target 205 based on this information, it does not reproduce from the reproduction position where the movement of the image virtual target 205 stops (the position when the image stops). The HMD 200 reproduces the image virtual object 205 from the reproduction position of the image virtual object 205 when the user 10 moves away from the front end of the line of sight of the image virtual object 205 or a predetermined time before the time of moving away from the front end of the line of sight. That is, the HMD 200 starts to reproduce the image virtual object 205 from the reproduction position of the image virtual object 205 before the time of moving away from the front end of the line of sight of the user 10 or a predetermined time before the time of moving away from the front end of the line of sight. As a result, the user 10 can also view the image reproduced from the time when the user 10 moves away from the image virtual object 205 to the time when the user 10 leaves the control section range 206.

[0089] Fig.13 This is a diagram for explaining an example of processing performed by the information processing device based on recorded mutual configuration information and information related to the user's position after approaching the predetermined distance and the direction of the user's line of sight when the user approaches within the predetermined distance from the paused image virtual target. That is, this is a diagram for explaining an example of a situation in which the information processing device reproduces the image virtual target from the reproduction position corresponding to the user's movement time from the user's stay position within the control segment range to the position that leaves the control segment range when reproducing the image virtual target. Fig.13 In the example, the user 10 is at the position 207 within the control section range 206, and sees and hears the movement of the image virtual target 205. Afterwards, as shown by arrow 1301, when the user 10 does not direct his / her sight toward the image virtual target 205 during the predetermined time and moves to the position 1302 and leaves the control section range 206, the movement of the image virtual target 205 stops. Furthermore, when the user 10 approaches the paused image virtual target 205, the HMD 200 reproduces the image virtual target 205 based on the recorded mutual configuration information and the information about the position of the user after approaching within the predetermined distance and the direction of the user's sight line.

[0090] However, at this time, when the action of reproducing the image virtual object 205 from the reproduction position where the user 10 moves the line of sight away from the image virtual object 205 is reproduced, there may be a case where the re-reproduction position is not the re-reproduction position that the user 10 considers to be preferred. In such a case, as an example, as the re-reproduction position that the user 10 considers to be preferred, consider the reproduction position when the user 10 starts moving from the position 207 to the position 1302. Therefore, the HMD 200 records the time until the user 10 leaves the control segment range 206 without directing the line of sight toward the image virtual object 205, and the time of the user movement 1303 from the user's staying position to the position leaving the control segment range 206. Furthermore, in a case where the user 10 moves to the position 1302 leaving the control segment range 206 without directing the line of sight toward the image virtual object 205 during a predetermined time, the HMD 200 reproduces from the reproduction position of the image virtual object 205 that is rewound by the user movement time.

[0091] Therefore, when the user 10 approaches within a predetermined distance from the paused image virtual object, the HMD 200 reproduces the image virtual object without interruption based on the recorded mutual configuration information and the information about the position of the user 10 after approaching within the predetermined distance and the direction of the line of sight of the user 10. Therefore, when reproducing the image virtual object, the user 10 can watch and hear the movement of the image virtual object 205 without interruption from the reproduction position of the image virtual object 205 watched and heard at the position 207.

[0092] In addition, the HMD 200 can determine the user's viewing and hearing importance for the image virtual object 205 based on information about the time until the user 10 leaves the control section range 206 without looking at the image virtual object 205. And, the HMD 200 can also appropriately select the reproduction position based on this determination. The HMD 200 can perform reproduction control such as selecting the rewind time such as rewinding the image virtual object 205 by the user's moving time or reproducing from the reproduction position when the user 10 moved the line of sight away from the image virtual object 205.

[0093] Next, use Fig.14 , describing an information processing device that lists a plurality of reproduction position candidates as the reproduction position of the image virtual target when approaching a predetermined distance from a paused image virtual target, selects a reproduction position based on the audiovisual importance of the reproduction position of the image virtual target, and notifies the reproduction position. Fig.14 in Figure 1 -Figure 5, Figure 7-Figure 13 The parts shown in the figure and attached with the same symbols have the same actions as those already described in the figure, so some of their detailed descriptions are omitted. Fig.14As shown, when the user 10 approaches the image virtual target in which the action is stopped and moves into the control segment range 206, in HMD200, multiple reproduction position candidates such as user positions 1401, 1402, and 1403 are selected as the reproduction position of the audiovisual image virtual target 205. Here, HMD200 displays a message image indicating the reason for selecting each reproduction position candidate and information related to the reproduction position of the image virtual target for notification. HMD200 generates images of virtual targets such as circles, arrows, and dots indicating the relationship between the image virtual target and the user's position in each reproduction position candidate and each position / direction (each audiovisual position / direction) of the user's line of sight and the gaze part, and displays them in a three-dimensional space for notification. Thereafter, the user 10 is located and faces any audiovisual position / direction of the selected reproduction position candidate, or further gazes at the gaze part. As a result, HMD200 resumes the reproduction of the image virtual target from the reproduction position corresponding to the audiovisual position / direction relationship and the reproduction position candidate of the gaze part.

[0094] In addition, candidates for the reproduction position are selected based on information such as the length / number of times the user 10 gazes, highlighted scenes (scenes with the largest number of views including information related to the viewing of other viewers), and the relationship between each viewing position / direction. For example, the starting reproduction position of a scene with a long gaze time of the user 10 is listed as a candidate together with information related to the viewing position / direction relationship. In addition, the information on the viewing position / direction of the reproduction position candidate is recorded as information on the position / direction with a predetermined range. By selecting candidates for the reproduction position in this way, when the user revisits important scenes when he or she starts viewing the virtual object of the image again, or when the viewing time is not very long, the viewing can be started again by omitting unimportant scenes.

[0095] Next, using FIG. 15, an information processing device is described in which a new fixed configuration of the image virtual target is performed after the user leaves the control section and the movement of the image virtual target is paused. In the example of FIG. 15, HMD 200 changes the position / direction of the image virtual target in accordance with the movement of HMD 200 while maintaining the position / direction relationship between the position of the image virtual target 205 and the user 10 and the line of sight of the user 10 during the pause. Various coordinate systems are used in the processing of representing three-dimensional physical objects and virtual targets. The three-dimensional world coordinate system is a coordinate system in which the result of geometrically modeling the real space and the virtual space is called the world and is assumed, and is a coordinate system in which the position and direction of the image virtual target 205 do not change even if HMD 200 moves. On the other hand, there is a coordinate system of the model itself suitable for representing each part of the three-dimensional model, and there is a local coordinate system of the three-dimensional model relative to the world coordinate system. In the local coordinate system of HMD 200, the image virtual object 205 is fixedly arranged in such a manner that the position and direction of the image virtual object 205 change in accordance with the movement of HMD 200 to follow the movement of HMD 200. FIG. 15 shows an example of a local coordinate system of HMD 200. That is, FIG. 15 shows a situation where the position and direction of the image virtual object 205 are changed in accordance with the movement of HMD 200 while maintaining the mutual position and direction relationship between the image virtual object 205 and the user 10 and the line of sight of the user 10 during the pause. In FIG. 15, Figure 1 -Figure 5, Figure 7-Figure 14 The parts shown in the figure and assigned the same symbols have the same actions as those already described in the figure, so some of their detailed descriptions are omitted.

[0096] FIG. 15( a ) shows a case where the user 10 leaves the control section range 206 and moves to the position 1602 as indicated by the arrow 1601, and a predetermined time further passes after the action of the image virtual object 205 is paused. In this case, the HMD 200 changes the position and direction of the image virtual object 205 in accordance with the action of the HMD 200 while maintaining the mutual position / direction relationship between the image virtual object 205 and the user 10 and the line of sight of the user 10 at the time of the pause. Furthermore, the HMD 200 notifies the user 10 whether to newly fix the image virtual object as indicated by the image virtual object 1603, and restarts the playback action.

[0097] In addition, in the notification at this time, HMD200 displays a selection screen on the display screen of HMD200, such as a screen that allows the user 10 to select whether to perform a new fixed configuration to the position at which the user 10 is looking, or to restart the reproduction action. Moreover, HMD200 performs a new fixed configuration and reproduction action of the image virtual object 205 according to the selection result of the user 10 who received the notification. That is, HMD200 notifies the user 10 whether to fix the image virtual object 205 to the local coordinate system of HMD200 and reproduce it, and controls the fixed configuration and reproduction according to the selection result of the user 10 in response to the notification. As a result, the user 10 can resume viewing and listening to the image virtual object 205 while maintaining the mutual position / direction relationship between the image virtual object 205 and the user 10 and the line of sight of the user 10 at the time of pause. Therefore, even if the user 10 does not return to the configuration location of the image virtual object 205, the user 10 can appropriately resume viewing and listening to the image virtual object 205.

[0098] Furthermore, when the image virtual target is fixedly configured in the local coordinate system of the HMD, if the size of the image virtual target 205 is greater than a predetermined threshold, the HMD 200 performs the processing described below. That is, as shown in FIG. 15( b ), the HMD 200 fixes the reduced image virtual target 1611 in the local coordinate system of the HMD 200 and displays it, and restarts the reproduction action of the image virtual target 1611. In addition, similarly, when the image virtual target is fixedly configured in the local coordinate system of the HMD, if the size of the image virtual target is greater than a predetermined threshold, the HMD 200 performs the processing described below. As shown in FIG. 15( c ), the HMD 200 fixes the image virtual target 1612 of the area of ​​the predetermined range including the gaze part of the image virtual target in the local coordinate system of the HMD 200 and displays it, and restarts the reproduction action of the image virtual target 1612. Therefore, it is possible to restart the viewing and listening of the reproduction action of the image virtual target while ensuring the field of view of the user 10 outside the image virtual target. Therefore, it is possible to stop or restart viewing with good usability through an appropriately timed operation according to the user's situation and the arrangement situation of the three-dimensional image virtual object.

[0099] Next, using FIG. 15( d ), the fixed arrangement and reproduction action of the image virtual object when the user leaves the control segment range when there are specific viewing places such as chairs and other viewers within the control segment range. FIG. 15( d ) shows a case where a chair 1621 as a specific viewing place and other viewers 11 are present at position 1624 within the control segment range 206 . In this case, when a predetermined time has passed since the user 10 moved out of the control segment range 206 , the HMD 200 copies the image virtual object while maintaining the position / direction relationship of the image virtual object and the user and the user's line of sight before the pause. Furthermore, the HMD 200 notifies the user 10 whether to newly fix the arrangement of the copied image virtual object 1623 and reproduces the action of the image virtual object 1623 . In addition, in the notification at this time, the HMD 200 displays a selection screen on the display screen of the HMD 200 so that the user 10 can select whether to perform a new fixed arrangement or restart the reproduction action by looking at it. Then, the HMD 200 controls a new fixed arrangement and a reproduction operation of the video virtual object 1623 according to the selection result of the user 10 who has received the notification.

[0100] Thus, by copying the image virtual object and reproducing it in a newly fixed configuration, the position and direction of the originally configured image virtual object are maintained as is, so that the user who has left the control section range can start viewing the image virtual object again in another place. That is, while considering the original viewing location of the image virtual object, the user who has left the control section can also view the image virtual object in another place before returning to the original configuration location.

[0101] In addition, when using the HMD, for example, there is a case where multiple users wearing HMDs view and listen to images of virtual targets (hereinafter sometimes referred to as image virtual targets) together. In such a case, it is considered that the image virtual targets are temporarily arranged in an appropriate space, and then the users view and listen. In addition, it is considered that the user wants to view the image virtual targets in the same way as when viewing the physical object, and the image virtual targets are displayed in an obstructive position on the HMD screen, and the same treatment is also considered.

[0102] In addition, when the user returns to the original position after starting to view the image virtual target again, HMD200 deletes the newly fixedly configured image virtual target 1623. In this way, only the original image virtual target 205 is fixedly configured, and confusion caused by unnecessary image virtual targets can be eliminated. Furthermore, in the case where there is a specific viewing and listening place such as a chair but there are no other viewers, HMD200 can also perform the processing described below. That is, HMD200 starts the reproduction again from the reproduction position obtained by shifting the reproduction time of the originally configured image virtual target 205 back by the amount of time for viewing the newly fixedly configured image virtual target 1632. In this way, the originally configured image virtual target and the newly fixedly configured image virtual target can be reproduced synchronously.

[0103] On the other hand, if there is a specific viewing location such as a chair or other viewers within the control section, when viewing is started again in that location first, the HMD 200 may be controlled so that the image virtual object is not fixedly arranged in the local coordinate system of the HMD and is not reproduced. This allows the user to quickly return to the location where the image virtual object is arranged without hindering the user's actions until the user returns to the location where the image virtual object is arranged.

[0104] In addition, regarding Figure 15(b) to Figure 15(d) The settings of various controls such as the above can be changed according to the user's preferences.

[0105] Next, the case where, when the user approaches other configurable places, a new image virtual object is fixedly arranged and the reproduction action is restarted while maintaining the position / direction relationship between the image virtual object and the user's position and the user's line of sight at the time of pause. FIG. 15(e) shows a case where the user 10 moves to position 1634 after leaving the control segment range 206 and moving to position 1631, and then approaches the truncated table 1633 as another configurable place within the control segment range 1632. Here, when the user 10 approaches the truncated table 1633 as another configurable place, as shown in FIG. 15(f), the HMD 200 performs the processing described below. While maintaining the position / direction relationship between the image virtual object 205 and the user's position and the user's line of sight at the time of pause, the HMD 200 identifies whether the image virtual object 1635 can be fixedly arranged on the truncated table 1633 as another configurable place. Here, when the HMD 200 identifies that the image virtual object 1635 can be fixedly arranged, it notifies the user 10 that the image virtual object 1635 can be fixedly arranged.

[0106] In addition, in the notification at this time, HMD200 will display a selection screen on the display screen of HMD200, such as a selection screen that allows the user 10 to select whether to perform a new fixed configuration or restart the reproduction action by looking at it. Moreover, HMD200 controls the new fixed configuration and reproduction action of the image virtual object 1635 according to the selection result of the user 10 who received the notification. Therefore, even if the user changes the configuration location of the image virtual object, the viewing and listening of the image virtual object can be restarted while maintaining the position of the image virtual object and the user before the pause, and the position / direction relationship of the user's line of sight. Therefore, it is possible to stop or restart the viewing and listening with good usability through appropriate timing actions corresponding to the user's situation.

[0107] In addition, the round table 1633 is an example of other configurable places, and other examples may also be used. In addition, the round table 1633 as another configurable place is shown to be within the control section range 1632, but it may also be outside the control section range 1632. In addition, when there are specific audio-visual places such as chairs and other viewers within the original control section range, HMD200 may also be controlled to notify that the image virtual target can be newly fixedly configured in other places and not to perform fixed configuration / reproduction control. As a result, it is possible to obtain an effect that a notification instruction and fixed configuration / reproduction control can be performed in consideration of the status of the original audio-visual place of the image virtual target.

[0108] use Fig.16 , describing a structural example of an information processing device according to this embodiment. Fig.16 : is a block diagram showing a configuration example of an HMD as an example of an information processing device according to the present embodiment. Fig.16 in Figure 1 , Figure 3 -Figure 5, Figure 7-Figure 1 5 and the same reference numerals are added to the parts having the same operation as that already described in the figure. Therefore, some of their detailed descriptions may be omitted. Fig.16 In the figure, HMD 200 is appropriately constructed by using a left eye sight sensor 201, a right eye sight sensor 202, a camera 203, a positioning sensor 215, a geomagnetic sensor 216, an acceleration sensor 1504, a gyro sensor 1505, an operation input interface 1507, a display processing device 1508, a processor 1520, a memory 1530 for storing programs 1531 and information data 1532, a vibration generating device 1541, a sound input device 1542, a sound output device 1543, and a communication device 1544, and each structural part is interconnected via a bus 1550.

[0109] The camera 203 is a device for capturing the field of view in front of the camera, and converts the light incident from the lens into an electrical signal using an imaging element to obtain a camera captured image. The camera 203 is appropriately set so as to be able to capture the field of view in front of the user 10. In an image see-through type HMD, the camera 203 captures a physical object in the real space such as a table 204, and the HMD 200 displays the captured image of the physical object on the display processing device 1508.

[0110] The left eye sight sensor 201 and the right eye sight sensor 202 can detect the movement and direction of the left eye and the right eye respectively, and capture the front of the sight of the user 10. In addition, the process of detecting the movement of the eyeball can be used by the well-known technology generally used as the eye tracking process. For example, in the method using corneal reflection, it is known that an infrared LED (Light Emitting Diode) is irradiated to the face and photographed with an infrared camera, and the position on the cornea of ​​the reflected light irradiated by the infrared LED (corneal reflection) is used as a reference point, and the movement of the eyeball and the sight line are detected according to the position of the pupil relative to the position of the corneal reflection.

[0111] As an example, the positioning sensor 215 has a GPS (Global Positioning System). It is a device that receives radio wave signals from artificial satellites to detect the current position, and can determine the current position of the user 10 wearing the HMD 200. In addition, as another example, a camera or a distance measuring sensor described later can also be used as a positioning sensor. By analyzing the image captured by the camera and the three-dimensional point group data measured by the distance measuring sensor, the current position can be obtained, and technologies such as SLAM (Simultaneous Localization and Mapping) are known. In addition, the geomagnetic sensor, acceleration sensor, and gyro sensor described later can also be used as positioning sensors. The current position can be obtained as a relative position from a reference position, and technologies such as PDR (Pedestrian Dead Reckoning) are known. In addition, Wi-Fi can also be used for positioning. By detecting the strength of each radio wave from multiple Wi-Fi access points and the difference in arrival time to perform three-point positioning, the current position can be obtained.

[0112] The geomagnetic sensor 216 is a sensor that measures the earth's magnetic field to detect the azimuth in which the user wearing the HMD 200 is facing. By using a three-axis type that detects geomagnetism in the up-down direction in addition to the front-back direction and the left-right direction, the geomagnetic change corresponding to the movement of the HMD 200 is captured, and the movement of the HMD 200 can also be detected.

[0113] The acceleration sensor 1504 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement / vibration / shock, etc. When the acceleration applied is only gravity, the acceleration sensor 1504 can use the gravity vector and its projection on the axis of the acceleration sensor to find the tilt angle, measure and detect how much it is tilted relative to the ground. The acceleration sensor 1504 provided in the HMD 200 can be used to detect the tilt of the HMD 200. In addition, the gyro sensor 1505 is a sensor that detects the angular velocity in the rotation direction, and can capture the state of the vertical / horizontal / oblique posture. The gyro sensor 1505 can measure and detect how much movement is in which direction. Therefore, the acceleration sensor 1504 and the gyro sensor 1505 can be used to detect the posture of the HMD 200, such as the direction.

[0114] The ranging sensor 1503 is a sensor that can measure the distance and angle to an object and capture the shape of an object such as an article in three dimensions. For example, a LiDAR (Light Detection and Ranging) sensor that irradiates an object with a laser such as an infrared ray and measures the returned scattered light is used to analyze and detect the distance to a distant object and the state of the object. A TOF (Time Of Flight) sensor that measures the reflection time of pulsed light irradiated to the subject for each pixel can also be used for ranging. A millimeter-wave radar that emits millimeter-wave radio waves and captures their reflected waves to detect the distance to the reflected object and the state of the object can also be used. The ranging sensor 1503 measures the distance and direction to a physical object such as a table 204, and the HMD 200 can, based on its measurement information, for example, confirm that the image virtual target 205 is configured on the table 204.

[0115] The processor 1520 is configured using an appropriate semiconductor device such as a CPU. The processor 1520 controls each component of the HMD 200 by executing an operating system (OS) 1533 and an application program 1534 for motion control stored in the memory 1530, thereby realizing functions such as the OS, middleware, and applications, and other functions. The HMD 200 includes a virtual object processing unit 1521, a reproduction control processing unit 1522, a mutual configuration information processing unit 1523, a predetermined gaze range processing unit 1524, an appropriate audiovisual configuration processing unit 1525, a reproduction position candidate processing unit 1526, a surrounding condition processing unit 1527, and a configuration availability processing unit 1528. These (1521 to 1528) are set as program modules executed by the processor 1520, and the HMD 200 controls the operation of each function through these (1521 to 1528).

[0116] The memory 1530 is composed of a nonvolatile storage device and the like, and stores various programs 1531 and information data 1532 processed by the processor 1520 and the like. As the information data 1532, virtual object information 1535, mutual configuration information 1536, predetermined viewing range information 1537, appropriate audiovisual configuration information 1538, configuration location surrounding information 1539, and the like are stored.

[0117] In the case of an optical see-through HMD, the display processing device 1508 can be configured to include a projection unit that projects virtual objects, notification information to the user, etc., and a transparent half-transparent half-reflective mirror that displays the projected virtual objects, etc., in front of the user's eyes. As a result, the user can visually recognize the virtual objects and the physical objects in the visual field in front of the user in the form of floating together. In addition, in the case of an image see-through HMD, it can be configured to include a display such as a liquid crystal panel that displays the physical objects and virtual objects in front of the user's eyes captured by the camera 203 in combination. As a result, the user 10 can visually recognize the physical objects and virtual objects in the visual field image in front of the user's eyes in an overlapping manner.

[0118] The operation input interface 1507 is an input unit composed of, for example, a keyboard, a key button, a touch key, etc., and is a structure for the user 10 to set and input the information that he / she wants to input. The operation input interface 1507 may be provided in the HMD 200 at a position and in a manner that allows the user 10 to easily perform input operations, or it may be separated from the main body of the HMD 200 and connected by wire or wirelessly. In addition, the HMD 200 may display an input operation screen in the display screen of the display processing device 1508, and obtain input operation information by the position on the input operation screen to which the line of sight is directed. In addition, the HMD 200 may display a pointer on the input operation screen, and obtain input operation information input by the user operating the pointer using the operation input interface 1507. In addition, the user 10 may make a sound indicating the input operation, and the HMD 200 may collect the sound using the sound input device 1542 to obtain the input operation information. In addition, a gesture corresponding to a predetermined operation may be registered, and the HMD 200 may obtain the user's operation action using the camera 203 or the like to obtain the input operation information.

[0119] The sound input device 1542 is a device that collects external sound and the user's own voice using a microphone and converts it into sound data. The HMD 200 can take instruction information consisting of the voice of the user 10 into the HMD 200 and perform an action based on the instruction information with good usability.

[0120] The sound output device 1543 can output sound from the speaker based on the sound data, and can use sound to report notification information to the user 10. For example, when the user 10 is located inside the control section range 206 or faces a position / direction different from the viewing position / direction before the pause, a sound is emitted to inform that the viewing position / direction of the user is different from the viewing position / direction of the user before the pause.

[0121] The vibration generating device 1541 vibrates under the control of the processor 1520, and converts the notification instruction information sent by the HMD 200 to the user 10 into vibration. The vibration generating device 1541 can vibrate the head of the user wearing the HMD 200 to let the user 10 know the notification instruction information, which can improve usability.

[0122] The communication device 1544 is a communication interface for wireless communication with other devices through short-range wireless communication. The communication device 1544 includes a communication processing circuit, an antenna, etc. corresponding to various predetermined communication interfaces, and sends and receives various information, control signals, etc. In addition, it may also include a telephone communication network.

[0123] The timer 1545 measures the time until the user 10 leaves the control section range 206 without directing the line of sight toward the image virtual target 205, the time of the user moving 1303 from the user's staying position to the position leaving the control section range, the elapsed time after the user 10 leaves the control section range 206, etc. The processor 1520 records the time measured by the timer 1545 in the memory 1530.

[0124] The virtual object processing unit 1521 generates a three-dimensional image virtual object 205 such as a football match image, and arranges it in a three-dimensional space. In addition, information related to the position and direction of the arranged image virtual object 205 is recorded in the memory 1530. In addition, when the user 10 is outside the control segment range 206 during a predetermined time, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 changes the position / direction of the image virtual object in accordance with the movement of the HMD 200 while maintaining the mutual position / direction relationship between the image virtual object and the user and the user's line of sight during the pause. In addition, the image virtual object is newly fixedly arranged in the local coordinate system of the HMD 200. In addition, when the image virtual object 205 is fixedly arranged in the local coordinate system of the HMD 200, if the size of the image virtual object is greater than a predetermined threshold value, the virtual object processing unit 1521 performs the processing described below. That is, the virtual object processing unit 1521 reduces the size of the video virtual object 205 or extracts and arranges the image virtual object 205 by extracting a region including a predetermined range of a fixation point for the video virtual object recorded by the predetermined fixation range processing unit 1524 described later.

[0125] In addition, when the user 10 reenters the control section range 206 after pausing the image virtual target, the virtual target processing unit 1521 performs the processing described below. That is, a virtual target is generated and displayed, which represents the position / direction relationship between the image virtual target 205 and the user 10, the mutual position / direction relationship of the user 10's line of sight, and the user 10's gaze position 232 for the image virtual target 205. For example, the virtual target processing unit 1521 generates an image of a virtual target such as a circle, an arrow, or a dot representing the above information and displays it in an overlapping manner in a three-dimensional space. In addition, an image of a virtual target such as a message image representing the reason for selecting the reproduction position candidate of the image virtual target and information related to the reproduction position of the image virtual target is generated and displayed in an overlapping manner in a three-dimensional space. Furthermore, an image of a virtual target such as a circle, an arrow, or a dot representing information related to the relationship between the image virtual target in the reproduction position candidate and each position / direction (each audio-visual position / direction) of the user is generated and displayed in an overlapping manner in a three-dimensional space.

[0126] The reproduction control processing unit 1522 pauses the movement of the image virtual object 205 when the user 10 moves away from the image virtual object 205 by a predetermined distance. Furthermore, when the user 10 approaches within a predetermined distance from the image virtual object 205, the reproduction control processing unit 1522 performs the processing described below. That is, the reproduction movement of the paused image virtual object 205 is controlled based on the recorded mutual arrangement information and the position of the user after approaching within the predetermined distance, the information related to the user's line of sight, the information related to the reproduction position candidate of the image virtual object 205, and the like. As an example of controlling the reproduction movement, it is possible to cite the reproduction of the image virtual object 205 as it is or to reproduce the image virtual object 205 by rotating it. In addition, based on the information related to the user's visual and auditory importance to the image virtual object and the visual and auditory importance of the reproduction position of the image virtual object, the image virtual object 205 is rewound to a predetermined time and reproduced. In addition, the reproduction control processing unit 1522 controls the reproduction / stop movement of the image virtual object 205 fixedly arranged in the local coordinate system of the HMD 200.

[0127] The mutual configuration information processing unit 1523 obtains the mutual position / direction relationship between the position of the image virtual object and the user and the user's line of sight before the image virtual object stops, generates mutual configuration information, and records it in the memory 1530. For example, as the mutual configuration information, the position / direction of the user 10 or the line of sight of the user 10 with respect to the image virtual object 205 at the stay position where the user 10 stays before leaving the control segment range 206 is recorded. Here, as an example, the stay position is the starting point position of the movement when the user 10 moves to the outside of the control segment range 206. The stay position can also be set to a position where it is determined that the change amount of the sensor value is smaller than the predetermined amount and the user 10 has not moved based on the information from various sensors (such as the positioning sensor 1503, the acceleration sensor 1504, and the gyro sensor 1505) at a predetermined time. In addition, when the user 10 does not move to the outside of the control segment range 206 in the direction of the image virtual object 205 during the predetermined time, the mutual configuration information processing unit 1523 performs the processing described below. That is, the mutual arrangement information processing unit 1523 records the location where the user 10 stayed before leaving the control section range 206 and the direction from the location to the video virtual object 205 .

[0128] Here, the direction from the stay position to the image virtual target 205 can be appropriately obtained. As an example, the direction of the HMD 200 displaying the image virtual target 205 can be detected by the acceleration sensor 1504 and the gyro sensor 1505 before stopping, and the detected direction of the HMD 200 can be recorded as the direction from the stay position to the image virtual target 205. In addition, the direction of the virtual target displayed in an overlapping manner can also be considered. In this way, the mutual configuration information processing unit 1523 uses the information related to the position of the user 10 relative to the image virtual target 205 and the line of sight of the user 10 obtained and recorded before the image virtual target 205 is paused, and records the mutual configuration information. That is, the mutual configuration information processing unit 1523 selects and obtains the audio-visual position / direction of the user 10 relative to the image virtual target 205 before the pause from the information, and records it as the mutual configuration information. In addition, the mutual configuration information can also be recorded as information of the position / direction having a predetermined range.

[0129] The predetermined gaze range processing unit 1524 selects the predetermined gaze range 233 centered on the user gaze area 232 before stopping determined by the left eye gaze sensor 201 and the right eye gaze sensor 202 to generate predetermined gaze range information and records it in the memory 1530. For example, when the position of the user 10's gaze line relative to the image virtual object 205 does not move during a predetermined time, the predetermined gaze range processing unit 1524 determines the area including the stationary gaze position as the gaze area 232. Alternatively, when the amount of movement of the user 10's gaze line relative to the image virtual object 205 during a predetermined time is less than a predetermined threshold, the predetermined gaze range processing unit 1524 determines the area including the moving position of the gaze line less than the predetermined threshold as the gaze area 232.

[0130] The appropriate audiovisual configuration processing unit 1525 selects the appropriate audiovisual position / direction of the image virtual target as the appropriate audiovisual configuration information, and records the selected appropriate audiovisual configuration information into the memory 1530. For example, as the appropriate audiovisual configuration information, the position and gaze point of the user before the pause can be cited. In addition, the appropriate audiovisual position / direction for the scene of the virtual object, etc., which the user is gazing at in the image scene of the image virtual target when pausing, etc. can be cited. In addition, there are appropriate audiovisual positions / directions for specific virtual objects that are gazed at in the image scene of the image virtual target when pausing, or that are pre-specified by the user. Here, for example, the gaze point, the virtual object being gazed at, etc. can be obtained based on the left eye sight sensor 201 and the right eye sight sensor 202. As for the direction to a specific virtual object, the direction can also be estimated by tracking the specific virtual object. The user's audiovisual position is a position where the gaze point, etc., which can be appropriately viewed and heard, can also be estimated based on the position of each virtual target.

[0131] The reproduction position candidate processing unit 1526 selects candidates for the image reproduction position for rewinding the image virtual target and reproducing it, and records it in the memory 1530. Candidates for the image reproduction position can also be selected based on information such as the length / number of times the user 10 gazes, the highlighted scene (including the scene with the most total viewing information including information related to the viewing of other viewers, etc.). In addition, candidates for the image reproduction position can also be selected based on information such as the relationship between the position of the image virtual target in the reproduction position candidate and the user, and the user's line of sight (viewing position / direction). In addition, in addition to such viewing conditions, candidates can also be selected based on the content of the image virtual target (for example, information related to a highlight tag indicating a highlight assigned to the image virtual target, information related to the action of an object focused on by the highlight, etc.).

[0132] The surrounding condition processing unit 1527 records the image captured by the camera 203 or displayed within the control section of the display processing device 1508 as the configuration location surrounding information 1539 in the memory 1530. In addition, the surrounding condition processing unit 1527 determines whether there are specific viewing locations such as chairs and other viewers around the image virtual object (within the control section) based on the recorded information. In addition, the objects determined as specific viewing locations such as chairs and other viewers can be controlled to be applied not only to real objects and people, but also to virtual objects different from the image virtual object.

[0133] The configuration possibility processing unit 1528 analyzes the distance measurement result using the distance measurement sensor 1503, the image captured by the camera 203 or displayed on the display processing device 1508. In addition, the configuration possibility processing unit 1528 recognizes the distance from the user 10 to the real object or other virtual target, the size of the real object or other virtual target, the presence or absence of a horizontal plane, the type (e.g., table, desk), etc., and determines whether the image virtual target can be configured.

[0134] With the above structure, HMD200 performs the following actions as basic actions. When the virtual object processing unit 1521 determines that the user has moved away from the three-dimensional image virtual object 205 arranged in the three-dimensional space by a predetermined distance, HMD200 uses the reproduction control processing unit 1522 to pause the action of the image virtual object. In addition, the mutual configuration information processing unit 1523 obtains and records the mutual position / direction relationship between the image virtual object and the user or the user's line of sight before the pause as mutual configuration information. After the pause, the user approaches within a predetermined distance from the image virtual object. At this time, the reproduction control processing unit 1522 controls the reproduction action of the image virtual object based on the recorded mutual configuration information and information related to the position of the user after approaching within the predetermined distance, the direction of the user's line of sight, and information related to the reproduction position candidate of the image virtual object 205. As the control of the reproduction action, the image virtual object is reproduced as it is, or the image virtual object is rotated and reproduced, or it is reproduced after a predetermined time.

[0135] In addition, in the HMD 200, the predetermined gaze range centered on the gaze part of the user is selected in the state before the pause, and is generated and recorded as the predetermined gaze range information by the predetermined gaze range processing unit 1524. After the pause, when it is recognized that the front end of the user's line of sight enters the predetermined gaze range indicated by the predetermined gaze range information, the reproduction action of the image virtual object is restarted.

[0136] In addition, in HMD 200, when the action of the image virtual object is stopped by the appropriate audio-visual configuration processing unit 1525, the appropriate audio-visual position / direction of the image virtual object is selected as the appropriate audio-visual configuration information, and the user is notified of the appropriate audio-visual configuration information selected. As the appropriate audio-visual configuration information, the position and gaze point of the user before the pause, and the appropriate audio-visual position / direction for the scene of the virtual object that the user is gazing at in the image scene of the image virtual object at the time of the pause can be cited. In addition, the appropriate audio-visual position / direction for a specific virtual object that is gazed at in the image scene of the image virtual object at the time of the pause or that is pre-specified can be cited.

[0137] In addition, in HMD 200, the image reproduction position at which the image virtual object is rewound and reproduced, and the candidate viewing and listening position / direction at the image reproduction position are recorded by using the reproduction position candidate processing unit 1526. The reproduction position candidate is selected and recorded based on the gaze time information such as the length of time and number of times the user gazes at the image virtual object, the information related to the highlighted scene (including the scene with the largest number of total viewings including information related to the viewing and listening of other viewers, etc.), and each viewing and listening position / direction. That is, the reproduction position candidate is selected and recorded based on the viewing and listening status of such an image virtual object and the content of the image virtual object (for example, information related to a highlight label indicating the highlight given to the image virtual object, information related to the action of the object focused on by the highlight, etc.). The reproduction action of the image virtual object is restarted from the image reproduction position selected by the user from the selected and recorded reproduction position candidates.

[0138] In addition, the virtual object processing unit 1521 determines that the user has moved away from a three-dimensional image virtual object 205 such as a football match image configured in a three-dimensional space by a predetermined distance. In addition, the reproduction control processing unit 1522 pauses the movement of the image virtual object, and the mutual configuration information processing unit 1523 obtains the position of the image virtual object and the user and the mutual position / direction relationship before the suspension of the line of sight and records them as mutual configuration information. Thereafter, when the user moves away from the image virtual object by a predetermined distance during a predetermined time, the following processing is performed in the HMD 200. That is, the virtual object processing unit 1521 fixes the position and direction of the image virtual object to the local coordinate system of the HMD in a manner that follows the movement of the HMD based on the mutual configuration information recorded by the mutual configuration information processing unit 1523. In addition, the reproduction control processing unit 1522 controls the reproduction / stop action of the image virtual object.

[0139] In addition, when the size of the image virtual object is larger than a predetermined threshold value when the image virtual object is fixedly arranged in the local coordinate system of the HMD by the virtual object processing unit 1521, the HMD 200 performs the following processing: that is, the image virtual object is reduced, or an area including a predetermined range of the gaze position for the image virtual object recorded by the predetermined gaze range processing unit 1524 is extracted and fixedly arranged in the local coordinate system of the HMD.

[0140] In addition, when the surrounding situation processing unit 1527 determines that there is a specific viewing location or other viewers around the arranged image virtual object, the HMD 200 performs the following processing: that is, the virtual object processing unit 1521 is used to control not to fix the image virtual object to the local coordinate system of the HMD, or to copy the image virtual object and fix it to the local coordinate system of the HMD.

[0141] In addition, the virtual object processing unit 1521 determines that the user is away from the three-dimensional image virtual object 205 such as a football match image configured in a three-dimensional space by a predetermined distance. In addition, the reproduction control processing unit 1522 pauses the action of the image virtual object, and the mutual configuration information processing unit 1523 obtains the mutual position / direction relationship between the position of the image virtual object and the user and the user's line of sight before the pause and records it as mutual configuration information. Thereafter, when the user approaches another configurable place and the configuration availability processing unit 1528 determines that a new image virtual object can be configured, the HMD 200 performs the following processing. That is, the virtual object processing unit 1521 controls to newly configure the image virtual object while maintaining the position of the image virtual object and the user and the mutual position / direction relationship between the user's line of sight at the time of the pause, so that the reproduction action is restarted.

[0142] Thus, an information processing device can be realized that can stop or restart viewing in a simple form with good usability by performing actions at appropriate timings corresponding to the user's situation and the situation of the arrangement of the three-dimensional image virtual object without requiring the user to perform special operations when viewing a three-dimensional image virtual object arranged in a specific place. For example, when a user wearing an HMD is viewing, even in the event of an emergency, the action of the image virtual object can be stopped or restarted with good usability at appropriate timing.

[0143] In the above description, HMD is used as a specific example of an information processing device, but all devices having the same function are targeted, and even information processing devices other than HMDs such as smartphones and smart watches can of course obtain the same functions and effects.

[0144] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, the above-mentioned embodiments are examples described in detail in order to explain the present invention in an easy-to-understand manner, and may not necessarily have all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of another embodiment can also be added to the structure of a certain embodiment. In addition, other structures can be added, deleted, or replaced with a part of the structure of each embodiment.

[0145] In addition, the numerical values, messages, etc. that appear in the text and drawings are merely examples, and the effects of the present invention will not be impaired even if different examples are used.

[0146] In addition, the above-mentioned structures, functions, processing parts, processing units, etc. can also be implemented in hardware by designing part or all of them with, for example, integrated circuits. In addition, the above-mentioned structures, functions, etc. can also be implemented in software by interpreting and executing programs that implement each function through a processor. The processor includes transistors and other circuits, which are regarded as circuits or processing circuits (processing circuitry). The information such as programs, tables, files, etc. that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, a DVD, or a device on a communication network.

[0147] In addition, regarding control lines and information lines, only those considered necessary for explanation are shown, and not all control lines and information lines necessary for the product are necessarily shown. It can also be considered that almost all the structures are actually connected to each other.

[0148] The user may freely set the range and shape of the control section. Alternatively, a method in which the user appropriately selects from a plurality of pre-registered patterns may be adopted. Information on the range of the control section determined by the user is stored in the memory 1530 and used for processing by the HMD 200 .

[0149] (Explanation of symbols)

[0150] 10: user; 200: head mounted display (HMD); 201: left eye sight sensor; 202: right eye sight sensor; 203: camera; 204: physical object (table); 205: image virtual target; 206: control section range; 208: display screen; 215: positioning sensor; 216: geomagnetic sensor; 1503: distance sensor; 1504: acceleration sensor; 1505: gyro sensor; 1507: operation input interface; 1508: display processing device; 1520: processor; 1521: Virtual target processing unit; 1522: Reproduction control processing unit; 1523: Mutual configuration information processing unit; 1524: Predetermined gaze range processing unit; 1525: Appropriate audio-visual configuration processing unit; 1526: Reproduction position candidate processing unit; 1527: Surrounding condition processing unit; 1528: Configuration feasibility processing unit; 1530: Memory; 1541: Vibration generating device; 1542: Sound input device; 1543: Sound output device; 1544: Communication device; 1545: Timer; 1550: Bus.

Claims

1. An information processing device, worn by a user, characterized in that: The information processing device comprises: A processor controls the reproduction of a three-dimensional image virtual target; A positioning sensor for detecting the position of the user; A sight sensor to detect the sight direction of the user; and Memory, The processor: recording in the memory a configuration position of the reproduced image virtual object and user information including at least one of the position of the user detected by the positioning sensor and the gaze direction of the user detected by the gaze sensor, When the positioning sensor detects that the user has left the range of a predetermined distance from the configuration position, the playback of the image virtual object is stopped. When the user enters the range after stopping the reproduction of the image virtual target, the reproduction of the image virtual target is controlled to start again based on the user information recorded in the memory and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor.

2. The information processing device according to claim 1, characterized in that When the user enters the range after stopping the reproduction of the image virtual target, the processor starts the reproduction of the image virtual target again if the position detected by the positioning sensor and the line of sight direction detected by the line of sight sensor correspond to the user information recorded in the memory.

3. The information processing device according to claim 1, characterized in that When the user enters the range after stopping the reproduction of the image virtual target, the processor controls in a manner to notify the user if the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory.

4. The information processing device according to claim 1, characterized in that The processor: When the amount of change in the user's line of sight direction detected by the line of sight sensor is below a threshold value during a predetermined time period, the range of the image virtual object in the line of sight direction is recorded in the memory as a gaze range, When the user enters the range after the reproduction of the image virtual object is stopped and the sight line direction of the user enters the gaze range, the reproduction of the image virtual object is started again.

5. The information processing device according to claim 1, characterized in that The processor: recording the position and orientation of the virtual object suitable for viewing and listening to the image into the memory as appropriate configuration information, When the user enters the range after the reproduction of the video virtual object is stopped, the user is notified of the appropriate arrangement information recorded in the memory.

6. The information processing device according to claim 1, characterized in that A display for displaying the image virtual object reproduced by the processor is provided, After stopping the reproduction of the image virtual target, when the user enters the range and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory, the processor controls the image virtual target to rotate and display it on the display.

7. The information processing device according to claim 1, characterized in that: With display, The processor: The image virtual target is controlled in such a way that it is overlapped with the physical object and displayed on the display, After the reproduction of the image virtual target is stopped, when the user enters the range and the position detected by the positioning sensor or the line of sight direction detected by the line of sight sensor is different from the user information recorded in the memory, the image virtual target is controlled to be rotated according to the shape or size of the physical object and displayed on the display.

8. The information processing device according to claim 1, characterized in that The processor controls the playback of the video virtual object to be resumed by returning to a position a predetermined time earlier than the position where the playback of the video virtual object was stopped when the user enters the range after the playback of the video virtual object is stopped.

9. The information processing device according to claim 1, characterized in that: The processor: recording a plurality of reproduction positions in the memory as positions for reproducing the image virtual object, When the user enters the range after the reproduction of the image virtual object is stopped, the user is notified of the plurality of reproduction positions. The control is performed so that the reproduction of the video virtual object is restarted from the reproduction position selected by the user.

10. The information processing device according to claim 1, characterized in that The processor: If the user does not enter the range even after a predetermined time has passed after the reproduction of the image virtual object is stopped, the user is notified whether to resume the reproduction of the image virtual object. When the user selects to resume playback, control is performed so that the video virtual object for which playback is resumed is arranged at a position different from the arrangement position recorded in the memory.

11. An information processing device having the function of generating and displaying a three-dimensional image virtual target, which is worn by a user, characterized in that: The information processing device includes a processor and a storage unit. The processor: recording, in the storage unit, mutual arrangement information which is a mutual positional direction relationship between the position of the image virtual object and the user and the line of sight, When the user leaves the image virtual object and the distance between the image virtual object and the user becomes greater than a predetermined distance, Stop the reproduction of the image virtual target, After a predetermined time has passed since the stop, the image virtual object is arranged so as to follow the movement of the information processing device in a state based on the mutual arrangement information recorded in the storage unit, Controlling the reproduction action of the configured image virtual object.

12. The information processing device according to claim 11, characterized in that: The processor: Notifying the user whether to reproduce the action of the configured image virtual object, When the user who has received the notification chooses to reproduce the action of the configured image virtual object, Execute the reproducing action of the configured image virtual object.

13. The information processing device according to claim 11, characterized in that: The processor: When it is recognized that the size of the image virtual object is larger than a predetermined threshold, the size of the image virtual object is reduced and arranged.

14. The information processing device according to claim 11, characterized in that: The processor: Before the stopping, the user's gaze position on the image virtual object is recorded in the storage unit, When it is recognized that the size of the image virtual object is larger than a predetermined threshold, the image virtual object is configured to be extracted in a range including a predetermined area of ​​the attention site recorded in the storage unit.

15. An information processing method for a device worn by a user, characterized in that: Reproduce three-dimensional image virtual targets, The reproduced image virtual target is configured and displayed in a manner overlapping with the physical object, recording a location where the image virtual object is configured and user information including at least one of the location of the user and the direction of the user's line of sight, When it is detected that the user leaves the range beyond the predetermined distance from the configuration position, the reproduction of the image virtual object is stopped, When the user enters the range after the reproduction of the image virtual object is stopped, the reproduction of the image virtual object is restarted according to the recorded user information.

Images