Information processing device, information processing system, information processing program, and information processing method

Abstract

To provide an information processing device that can improve the user experience when initiating interaction with video content. [Solution] The information processing device includes an acquisition means for acquiring the position of the user's hand, a control means for controlling the placement of image content in a virtual space, and a determination means for determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein the determination means does not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject.

Description

Technical Field

[0004] ,

[0006] , , , , ,

[0005] , , , , ,

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] A cross-reality (XR) system that allows a user to experience virtual reality using a head-mounted display (HMD) has been proposed. Such an XR system is used for viewing videos, and in recent years, spatial videos that can be viewed as stereoscopic videos have become popular. For example, Patent Document 1 discloses a technique for displaying a second virtual space on a first virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, Patent Document 1 is a technique for displaying different virtual spaces, and interference by extending a hand into the virtual space is not considered. For example, even if video content (image content) such as a spatial video is played in a window instead of the second virtual space and a hand is extended to interfere with an object shown in the video content, providing such tactile feedback is not considered. If it becomes possible to interfere with an object shown in the video content, an improvement in the user experience can be expected.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to improve the user experience when interfering with video content.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing device comprising: an acquisition means for acquiring the position of a user's hand; a control means for controlling the placement of image content in a virtual space; and a determination means for determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein the determination means does not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject. [Effects of the Invention]

[0007] According to the present invention, the user experience when interacting with video content can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating an information processing system according to the first embodiment of the present invention. [Figure 2] This diagram illustrates the internal configuration of an information processing device and a user assistance device according to a first embodiment of the present invention. [Figure 3] This is a schematic diagram illustrating an example of a scene that interferes with image content according to the first embodiment of the present invention. [Figure 4] This is a flowchart illustrating the determination of interference with a virtual object according to the first embodiment of the present invention. [Figure 5] This is a schematic diagram illustrating an example of a scene in which a user's hand touches a virtual object in a virtual space or content in a spatial video according to the first embodiment of the present invention. [Figure 6] This figure illustrates an example of a method for displaying a user's hand according to the first embodiment of the present invention. [Figure 7] This figure illustrates a specific example of determining a virtual object that a user's hand can interact with, according to the first embodiment of the present invention. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted in each drawing. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In the following embodiments, the same components will be denoted by the same reference numerals.

[0010] (First embodiment) <Configuration of the Information Processing System> Referring to Figure 1, the information processing system according to Example 1 will be described. The information processing system includes an HMD 100 and a user assistance device 110.

[0011] The HMD100 is a display device (electronic device) that can be worn on the user's head. The HMD100 is equipped with a camera to capture images in front of the user and a display to show the images to the user. The HMD100 displays a composite image on the display that fuses the image captured by the camera with virtual objects, which are computer graphics (CG) content. This allows the user to experience virtual reality with their own eyes. The HMD100 also has a function (hand tracking) that detects the user's hands from the captured images and acquires information about the position and posture of the hands, allowing the hand movements to interact with the virtual objects. This allows the user to intuitively manipulate virtual objects using their hands.

[0012] In the first embodiment, an HMD is assumed as an example of a device constituting an information processing system, but the present invention is not limited to these embodiments. For example, it may be a smartphone or tablet terminal equipped with a camera, or other devices such as a personal computer (PC) or digital camera.

[0013] The user assistance device 110 is a device for presenting the tactile force sensation instructed from the HMD 100 to the user. In the first embodiment, the shape of the user assistance device 110 will be described as a glove type, but it may be a bracelet type, a ring type, or the like.

[0014] <Internal configuration of HMD100> Referring to Fig. 2(a), the internal configuration of the HMD 100 will be described. The HMD 100 includes a control unit 200, a communication unit 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an imaging unit 204, and a display unit 205.

[0015] The control unit 200 controls each component of the HMD 100. The control unit 200 executes a program stored in the ROM 203, utilizes the RAM 202 as a work area, and controls the processing unit of the entire HMD 100. The control unit 200 includes at least one CPU (Central Processing Unit) that executes the program stored in the ROM 203 and at least one other circuit. Note that the control unit 200 may be composed of one or more processors such as a CPU or a GPU. Note that instead of the control unit 200 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0016] The communication unit 201 is an interface for directly communicating with the user assistance device 110 (communication unit 211) or an external device, or communicating via the Internet to perform transmission and reception of various data such as files and commands. Note that wireless communication may be performed using a wireless communication method such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark), or wired communication may be performed.

[0017] The RAM 202 is a temporary storage area such as the main memory and work area of the HMD 100. The RAM 202 stores the spatial coordinates and hand display method used for hand contact determination, which are determined based on the flowchart of FIG. 4. The RAM 202 consists of, for example, a volatile memory (DRAM) using semiconductor elements.

[0018] The ROM 203 is an electrically erasable and recordable non-volatile memory. When the HMD 100 is powered on, the control unit 200 reads a program from the ROM 203 and starts controlling the HMD 100. The ROM 203 is, for example, a large-capacity NAND-type flash memory, and stores a control program executed by the control unit 200 and video data of spatial video. Spatial video is image content that reproduces the depth relationship of the subject at the time of shooting. Although it is expressed as spatial video, it may be still image content.

[0019] Here, the control unit 200, RAM 202, and ROM 203 have been described as individual hardware resources, but these functions may be integrated and realized on one LSI.

[0020] The imaging unit 204 includes two cameras (imaging devices). The two cameras are arranged near the positions of the user's left and right eyes when the HMD 100 is worn in order to image a space similar to the space the user normally sees. Monochrome or color images (captured images) captured by the two cameras of the subject (the range in front of the user) are output to the RAM 202. Also, the control unit 200 acquires information on the distance to the subject as distance information by distance measurement using a stereo camera. The imaging unit 204 is composed of, for example, an optical lens unit and an optical system that controls aperture, zoom, focus, etc., and an imaging element for converting light (video) introduced through the optical lens unit into an electrical video signal. As the imaging element, generally, a CMOS imaging element (CMOS image sensor) using CMOS or a CCD imaging element (CCD image sensor) using CCD is used.

[0021] The display unit 205 displays a composite image of the captured image and virtual objects, as well as operation menus for controlling the HMD 100, as a three-dimensional image. The display unit 205 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, an organic EL panel is positioned in front of each of the user's eyes. It is also possible to use a device with a semi-transparent half-mirror for the display unit 205. In this case, for example, the display unit 205 may display an image so that virtual objects are superimposed directly onto the real space visible through the half-mirror, using a technology generally known as AR (Augmented Reality). Alternatively, the display unit 205 may display an image of a completely virtual space without using captured images, using a technology generally known as VR (Virtual Reality).

[0022] The attitude sensor unit 206 acquires attitude (and position) information of the HMD 100. The attitude sensor unit 206 then acquires attitude information of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 206 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 206 is used when acquiring user attitude information, and the control unit 200 controls it to acquire user attitude information.

[0023] Here, HMD100 is described as an example of a head-mounted display device with a built-in information processing device, but the information processing device is not limited to this. The information processing device connected to the head-mounted display device by wire or wireless means may be a smartphone, tablet terminal, or PC. In this case, some of the processing performed by the HMD100 described above may be performed by the information processing device connected to the head-mounted display device by wire or wireless means.

[0024] <Internal configuration of user assistance device 110> The internal configuration of the user assistance device 110 will be explained with reference to Figure 2(b).

[0025] The user assistance device 110 includes a control unit 210, a communication unit 211, and a haptic feedback unit 212.

[0026] The control unit 210 is a CPU that controls each component of the user assistance device 110. The control unit 210 controls the haptic feedback display unit 212 according to the haptic feedback parameters obtained from the HMD 100 via the communication unit 211. Alternatively, instead of the control unit 210 controlling the entire device, multiple hardware components may share the processing to control the entire device. Alternatively, instead of the control unit 210 controlling the user assistance device 110, the control unit 200 of the HMD 100 may control the user assistance device 110.

[0027] The communication unit 211 communicates with the HMD100 (communication unit 201).

[0028] The tactile feedback unit 212 has a motor that provides tactile feedback information and, according to the provided tactile feedback parameters, provides the user with a desired feel or sensation of force, object, weight, movement, or shape change. The tactile feedback parameters are not limited to the force of mass; tactile effects based on information about the physical characteristics of the virtual object may also be provided. The physical characteristics may include at least one of the following: size, shape, stiffness, weight, thermal characteristics, or texture of the virtual object.

[0029] The attitude sensor unit 213 acquires attitude (and position) information of the user assistance device 110. The attitude sensor unit 213 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The control unit 210 acquires the attitude (and position) information of the user assistance device 110 and controls it to transmit it to the control unit 200 via the communication unit 211. Note that the user assistance device 110 does not necessarily have an attitude sensor unit 213; in that case, the control unit 200 of the HMD 100 acquires (estimates) the attitude (and position) information of the user assistance device 110 based on the captured image taken by the imaging unit 204.

[0030] <Usage Example of HMD and User Assistance Device> Referring to FIG. 3, a usage example of the information processing system in Embodiment 1 of the present invention will be described. FIG. 3 is a view of the user from directly above. In FIG. 3, it is assumed that the user is wearing the HMD 100 and the user assistance device 110 and experiencing virtual reality (virtual space).

[0031] In the virtual space, there are virtual objects and operation menus, and their three-dimensional position coordinates are included in the space coordinates 300. The space coordinates 300 are used, for example, to act on virtual objects and operation menus by hand tracking.

[0032] The user's visual field 301 is the visual field range visible to the user and is the range of the video projected on the display unit 205.

[0033] The space video window 302 is a virtual object for viewing a space video. In FIG. 3, the space video window 302 is arranged within the range of the user's visual field 301. A space video is projected onto this space video window 302 to view the video.

[0034] ]> The space coordinates 303 of the space video represent the feature points of the subject included in the video of the space video in three-dimensional position coordinates. The control unit 200 acquires based on the distance information obtained by measuring the stereo camera video of the space video.

[0035] The user viewing range 304 of the space video is the video range visible to the user among the space video images. The visible range varies according to the user's viewing position, and when the user moves, the range of the video projected onto the space video window 302 changes.

[0036] <Processing Procedure of HMD> Referring to the flowchart in Figure 4, the processing procedure of the information processing system in Embodiment 1 of the present invention will be explained. This flowchart starts when a virtual object is displayed in the virtual space and is executed at least the first time. Note that the timing of execution of this flowchart is not limited to the timing when a spatial video window is displayed in the virtual space. For example, it may be when the user starts the HMD or when the user starts an application on the HMD. Note that the following flowchart may be executed in response to a reset operation performed by the user at a predetermined timing. Note that the following flowchart may be executed not only before the spatial video window is placed, but also after the spatial video window is placed.

[0037] In step S401, the control unit 200 starts displaying the virtual space on the display unit 205 in response to the user's operation to start using the information processing system. The control unit 200 switches the spatial coordinates used for contact detection (interference detection) under predetermined conditions. Initially, the spatial coordinates used for contact detection are stored in the virtual space spatial coordinates 300 and RAM 202. The control unit 200 also switches the display method of the user's hand under predetermined conditions. For example, it is possible to switch between a format that displays the entire hand and a format that makes the hand semi-transparent, but other display methods may also be used. Immediately after startup, the hand is stored in the format that displays the entire hand.

[0038] In step S402, the control unit 200 controls the imaging unit 204 to capture an image of the user's surroundings. The control unit 200 detects the user's hand from the captured image and acquires information regarding the position and orientation of the user's hand in the virtual space.

[0039] In step S403, the control unit 200 obtains the position of the spatial video window 302 in the virtual space. If the spatial video window 302 is not already placed in the virtual space, the control unit 200 places the spatial video window 302 and obtains its position.

[0040] In step S404, the control unit 200 determines whether the user's hand has come into contact with a virtual object using the spatial coordinates stored in the RAM 202. That is, the control unit 200 performs contact detection between the user's hand and a virtual object. Examples of virtual objects include the spatial video window 302, subjects included in the video displayed in the spatial video window 302, and virtual objects other than the spatial video window 302. In this flow, when contact detection is performed using the spatial coordinates of the spatial video, contact detection is performed with subjects shown in the spatial video, and contact detection is not performed with virtual objects other than virtual objects related to the spatial video. Similarly, when contact detection is performed using the spatial coordinates of the virtual space, contact detection is performed with virtual objects other than virtual objects related to the spatial video, and contact detection is not performed with subjects shown in the spatial video. In other words, while contact detection is performed using the spatial coordinates of the spatial video, even if the user touches (interferes with) a virtual object such as an interior piece placed in the virtual space, it will not be determined as contact. Likewise, while contact detection is performed using the spatial coordinates of the virtual space, even if the user touches (interferes with) a subject shown in the spatial video, it will not be determined as contact.

[0041] Referring to Figure 5(a), we will now explain a scenario in which contact detection is performed using spatial coordinates 300 in the virtual space, and the user's hand interferes with a virtual object in the virtual space. In Figure 5(a), there are virtual objects 500 other than the spatial video window 302 in the virtual space, and they are included in the field of view 301 of the HMD 100. Also, if the spatial video window 302 is an opaque screen, the field of view 304 is the range beyond the spatial video window 302 that cannot be seen. The spatial coordinates 300 also include coordinates based on the virtual object 500. When the user reaches out to touch the virtual object 500, the position of the user's hand approaches the spatial coordinates that represent the virtual object 500, and interference (contact) occurs. In step S404, if the control unit 200 detects interference, it may send a control signal to the user assistance device 110 to generate haptic feedback corresponding to the virtual object 500 and provide it to the user.

[0042] Next, referring to Figure 5(b), a scenario in which a user's hand interferes with content contained in a spatial video by performing contact detection using the spatial coordinates 303 of the spatial video is described. The spatial video content 501 is the video content contained in the spatial video image. The spatial coordinates 303 of the spatial video also include coordinates based on the spatial video content 501. When a user extends their hand into the spatial video window 302 and reaches out to touch the spatial video content 501, the position of the user's hand approaches the spatial coordinates that represent the spatial video content 501, causing interference. When the control unit 200 detects interference, it sends a control signal to the user assistance device 110, which generates haptic feedback corresponding to the spatial video content 501 and provides it to the user. Contact detection is performed even if the spatial video content 501 is located outside the user's spatial video field of view 304. Furthermore, a method is disclosed for predicting the position of the user's hand a few seconds later from its movement. This method may be used to sense the future position of the user's hand and prepare in advance to switch the spatial coordinates used for contact detection.

[0043] In step S405, the control unit 200 displays the hand according to the hand display method stored in the RAM 202. Figures 6(a) and 6(b) show examples of how the user's hand is displayed when the user is reaching into the spatial video window. Figure 6(a) is an example where the spatial video window 302 is viewed from the front, slightly to the right. The user's hand 600 is the user's hand in the virtual space, reaching into the spatial video window 302. The hand 601 is the part that extends into the spatial video window and is displayed semi-transparently. In AR and MR (Mixed Reality) spaces, one possible method is to overlay computer graphics on top of the real-world image of the user's hand and apply blurring or mosaic effects. Other possible methods include changing the hand's color to, for example, black and white, acquiring background images behind the hand from other camera images, or creating and overlaying pseudo-background images behind the hand from images around the hand, and displaying them semi-transparently (processing to increase transparency) or invisible. In a VR space, possible methods include blurring or mosaic processing the CG image of the avatar corresponding to the user's hand, changing the CG image of the hand to semi-transparent or another color, and hiding the CG image of the hand. Also, Figure 6(b) shows an example of viewing the spatial video window 302 from the front. In this case, the hand 600 is reaching out to the spatial video window 302, but the hand does not extend beyond the spatial video window 302, so the hand is displayed as is.

[0044] In particular, consider the case of hand display methods (when switched to hidden or semi-transparent) as described in step S408 below. As shown in Figure 6(a), if the back of the spatial video is visible when the hand is reaching into the spatial video, the hand will be visible behind the spatial video in the MR space. Originally, the spatial video's image space extends beyond the spatial video window, so it can be detrimental to the user experience if the hand appears to be sticking out of the spatial video window when it is reaching into the spatial video. Therefore, if the back of the spatial video is visible when the hand is reaching into the spatial video, computer graphics are overlaid on the real-world image of the user's hand, and blurring or mosaic processing is applied to hide the hand sticking out of the spatial video window. This prevents the hand from appearing to be sticking out behind the spatial video window, improving the user experience. Note that, as shown in Figure 6(b), if the spatial video is viewed from the front and the hand is contained within the spatial video, the hand is displayed without blurring or mosaic processing to hide it. Additionally, parts of the user's hands and arms that appear to be within the spatial video will be displayed as normal, while parts that extend beyond the spatial video will be blurred or pixelated to conceal the hands.

[0045] In step S406, the control unit 200 determines whether the user's hand moved into the spatial video area by passing through the frame (the edge of the window surface) without touching the outer frame (the edge of the window surface) of the spatial video window. If the control unit 200 determines that the user's hand moved into the spatial video area by passing through the frame without touching the outer frame of the spatial video window, the process proceeds to step S407. If the control unit 200 does not determine that the user's hand moved into the spatial video area by passing through the frame without touching the outer frame of the spatial video window, the process proceeds to step S409. In this embodiment, the outer frame of the spatial video window is used, but it may also be the edge of a predetermined surface. That is, it is not limited to the outer frame of the spatial video; a frame (edge ​​of a surface) for making the above determination may be provided. Furthermore, such a frame (edge ​​of a surface) may or may not be visible to the user.

[0046] In step S407, if the control unit 200 determines that the user's hand has passed by the method in step S406, it switches the spatial coordinates used for contact detection stored in RAM 202 to the spatial coordinates 303 of the spatial video. That is, it determines interference with subjects shown in the spatial video, and does not determine interference with virtual objects other than the virtual objects related to the spatial video.

[0047] In step S408, if the control unit 200 determines that the user's hand has passed by the method in S406, it switches the display method of the user's hand stored in RAM 202 to blur, mosaic, semi-transparent, or hide.

[0048] In step S409, the control unit 200 determines whether the user's hand has moved out of the spatial video window 302, regardless of whether the user's hand has touched the outer frame of the spatial video window. If the user's hand does not interfere with the spatial video window 302, the control unit 200 determines that the user's hand is outside the frame. If the control unit 200 determines that the user's hand has moved out of the spatial video window 302, regardless of whether the user's hand has touched the outer frame of the spatial video window, the process proceeds to step S410. If the control unit 200 does not determine that the user's hand has moved out of the spatial video window 302, regardless of whether the user's hand has touched the outer frame of the spatial video window, the process proceeds to step S412.

[0049] In step S410, if the control unit 200 determines that the user's hand is outside the frame by the method in step S409, it switches the spatial coordinates used for contact detection stored in RAM 202 to the spatial coordinates 300 in the virtual space. That is, it determines interference with a virtual object different from the virtual object related to the spatial video, and does not determine interference with a subject that appears in the spatial video.

[0050] In step S411, if the control unit 200 determines that the user's hand is outside the frame by the method in step S409, it switches the display method of the user's hand stored in RAM 202 to a normal display without blurring or mosaic effects.

[0051] In step S412, the control unit 200 determines whether or not to terminate the system. If the control unit 200 determines to terminate the system, it terminates this flow; otherwise, it proceeds to step S402.

[0052] The HMD processing procedure has been explained above with reference to the flowchart in Figure 4. According to the above flow, contact with a subject in the spatial video is determined only when the user's hand first interferes with the display surface of the spatial video window and moves into the spatial video area by passing through the frame without touching the spatial video window frame. Also, if the hand is extended into the spatial video but appears to protrude into the background of the spatial video, the display is changed to hide the part of the hand that protrudes into the background of the spatial video. Furthermore, if the user first touches the spatial video window frame before moving into the spatial video area, even if they interfere with a subject in the spatial video, it will not be determined that they have interfered with the subject. Here, the timing of first touching the spatial video window frame refers to the first timing in the time interval in which the user is interfering with the virtual object related to the spatial video. In other words, the timing when the user stops interfering with the spatial video window after being in a state of interfering with the spatial video window is the timing of the end of the time interval in which the user is interfering with the virtual object related to the spatial video. After the HMD is started, the user will interfere with the spatial video window, stop interfering with the spatial video window, and then start interfering with the spatial video window again, at which point the timing of first touching the spatial video window frame will occur again.

[0053] The above flow describes a flow in which, in addition to providing haptic feedback when contact is detected, the method of displaying the hand is also switched. However, this is not the only way. The method of displaying the hand may not be changed, and haptic feedback may be provided instead. In other words, steps S405, S408, and S411 may be omitted. Alternatively, the method of displaying the hand may be switched without providing haptic feedback. In other words, steps S404, S407, and S410 may be omitted.

[0054] In the above-described flow, the process is switched based on whether or not the spatial video window frame is touched, but this is not the only option. For example, it is not limited to spatial video; it could also be planar video content, planar image content, or a three-dimensionally displayed image. Alternatively, instead of the spatial video window frame, a predetermined surface used for the above determination could be defined, and the process could be switched based on whether or not the edge of the predetermined surface is touched first.

[0055] <Specific examples of determining virtual objects that can be interacted with by hands> In Figure 6(a) above, the user extends their hand into the center of the spatial video window 302 without touching its outer frame, and the hand passes through to the other side of the spatial video window 302. In this state, interference with a subject inside the spatial video window is detected. The portion of the hand that has passed through to the other side of the spatial video window 302 is considered to be in a "passing" state, and processing such as transparency (processing to increase transparency), blurring, and mosaic are applied to the portion of the hand that has passed through, as described above.

[0056] The following describes specific examples of how to determine virtual objects that a hand can interact with, referring to Figures 7(a), 7(b), and 7(c). Specifically, depending on the type of image content, we will explain cases where interaction is possible with virtual objects related to the image content, and cases where interaction is possible with virtual objects different from those related to the image content.

[0057] Figure 7(a) describes how the user's hand is displayed when it first touches the outer frame of the spatial video window 302 in the user's field of view 301. When the user's hand first touches the outer frame of the spatial video window 302, even if the hand passes to the other side of the spatial video window 302, the hand is not considered to have passed through. That is, the hand is displayed without applying processing such as transparency, blurring, and mosaic. Furthermore, various methods can be considered for determining whether the hand has passed through, and examples of other implementation methods are shown in Figures 7(b) and 7(c).

[0058] Figure 7(b) assumes a scenario in which a spatial video window 702 and a subject 701 floating in front of the spatial video are placed in the virtual space within the user's field of view 301. The subject 701 is smaller in shape than the spatial video window 702 and is placed adjacent to it. In this case, if the user's hand first touches the spatial video window 702 and then touches the subject 701, it is determined that the user has touched the subject 701. That is, if the user's hand first touches the spatial video window 702 and then touches the subject 701, the hand passing through the other side of the spatial video window 702 is considered to be in a passing state.

[0059] Figure 7(c) assumes a scenario in which a spatial video window 703 and a subject 704 positioned behind the spatial video are placed in the virtual space within the user's field of view 301. The spatial video window 703 is positioned in front of the subject 704 and has a donut-shaped hollow frame, for example, a square shape. In this case, if the user's hand first touches the spatial video window 703 and then touches the subject 704, it is determined that the hand has touched the subject 704. That is, if the user's hand first touches the spatial video window 703 and then touches the subject 704, the hand passing behind the spatial video window 703 or the subject 704 is considered to be in a passing state.

[0060] (modified version) In the first embodiment described above, an example was explained in which, if the user's hand first interferes with the frame of the spatial video window, even if it subsequently interferes with a subject in the spatial video, it is not determined that the subject has been interfered with. However, the system may also choose not to determine that the subject has been interfered with if the user's hand first interferes with the right or left side of the frame of the spatial video window. In other words, if the user's hand first interferes with the upper or lower side of the frame of the spatial video window, it may be determined that the subject has been interfered with if it interferes with a subject in the spatial video. When the spatial video is placed close to the user, when the user raises their hand from a lowered position to begin interfering with the spatial video, they may initially touch the lower side of the frame of the spatial video even though they intend to interfere with the spatial video. In such cases, even if the subject is interfered with, it will not be determined that the subject has been interfered with, so the user may fold their hand and raise it, or move a predetermined distance away from the spatial video before interfering with it, which may impair the user experience. Therefore, as described above, if the user's hand first interferes with the lower side of the frame of the spatial video window, it may be determined that the hand has interfered with the subject shown in the spatial video. Furthermore, if the distance between the user (HMD) and the spatial video window is less than a threshold, interference with the subject shown in the spatial video may be determined even if the user's hand first interferes with the upper or lower side of the frame of the spatial video window. Also, if the distance between the user (HMD) and the spatial video window is not less than a threshold, if the user's hand first interferes with the frame of the spatial video window, it may be determined that the hand has not interfered with the subject shown in the spatial video, even if it subsequently interferes with the subject. In other words, the frame used as a trigger to prevent the detection of hand interference with a subject does not have to consist of four sides like a rectangle, but may consist of at least one side.

[0061] Furthermore, the first embodiment described above does not explain how to display the spatial video when the user's hand first interferes with the frame of the spatial video window, nor does it specifically mention changing the display method. However, the display method of the spatial video may be changed when the user's hand first interferes with the frame of the spatial video window, i.e., as shown in Figure 7(a). For example, if the user's hand first interferes with the frame of the spatial video window, even if it interferes with the subject of the spatial video, it will not be determined that it has interfered with the subject, so the transparency of the spatial video may be increased. Alternatively, instead of increasing the transparency, the spatial video may be hidden. In this way, the user can visually recognize that they cannot interfere with the spatial video, and if they want to interfere with the spatial video, they can try again. Also, if the user's hand first interferes with the inside of the display surface of the spatial video window, the transparency of the spatial video will be maintained and displayed without increasing its transparency. Note that the frame of the spatial video and the display surface are just examples, and a predetermined surface used for the above determination may be defined, and the processing may be switched depending on whether the user first touches the edge of the predetermined surface or the inside of the predetermined surface. The predetermined surface is a virtual display that displays the image content, and the edges of the predetermined surface may be the edges of the image area of ​​the virtual display.

[0062] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0063] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0064] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0065] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0066] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0067] This embodiment includes the following configurations, methods, and programs.

[0068] [Configuration 1] A means for obtaining the position of the user's hand, A control means for controlling the placement of image content in a virtual space, The system includes a determination means for determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination means will not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject. An information processing device characterized by the following:

[0069] [Configuration 2] The determination means does not determine whether the hand and the subject have interfered if the hand first interferes with the edge when it begins to interfere with the image content. The information processing device according to configuration 1, characterized by the above.

[0070] [Configuration 3] The determination means determines whether the hand has interfered with the subject if, when the hand begins to interfere with the image content, it does not initially interfere with the edge of the image content. An information processing apparatus according to configuration 1 or 2, characterized by the above.

[0071] [Structure 4] The determination means determines that interference between the hand and the subject has occurred if the hand first interferes with the upper or lower edge when it begins to interfere with the image content, but does not determine that the hand has interfered with the subject, even if the hand interferes with the subject, if the hand first interferes with the right or left edge when it begins to interfere with the image content. An information processing device according to any one of configurations 1 to 3.

[0072] [Composition 5] The determination means determines that, when the distance between the user and the image content is less than a threshold, if the hand first interferes with the upper or lower edge when it begins to interfere with the image content, then interference between the hand and the subject is determined; however, if the hand first interferes with the right or left edge when it begins to interfere with the image content, then even if the hand interferes with the subject, the determination means does not determine that the hand has interfered with the subject. An information processing apparatus according to any one of configurations 1 to 4, characterized by the above.

[0073] [Composition 6] The determination means further comprises a generation means for generating haptic feedback when it is determined that the hand and the subject have interfered with each other. An information processing device according to any one of configurations 1 to 5, characterized by the above.

[0074] [Composition 7] If the determination means determines that the hand and the subject have interfered with each other, the display of the subject is changed. An information processing device according to any one of configurations 1 to 6.

[0075] [Structure 8] The subject is a subject located behind the image content. An information processing device according to any one of configurations 1 to 7, characterized by the above.

[0076] [Composition 9] The acquisition means acquires the position of the hand based on the captured image. An information processing device according to any one of configurations 1 to 8.

[0077] [Configuration 10] The aforementioned image content is an image content that reproduces the depth relationship of the subject at the time of shooting. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above.

[0078] [Composition 11] The control means controls the transparency of the image content to be increased or the image content to be hidden if the hand first interferes with the edge of a predetermined surface relating to the image content when it begins to interfere with the image content. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above.

[0079] [Composition 12] The control means maintains the transparency of the image content if, when the hand begins to interfere with the image content, it first interferes with the inside of a predetermined surface relating to the image content. The information processing apparatus according to configuration 11, characterized by the features described above.

[0080] [Composition 13] The predetermined surface is a virtual display that displays the image content, and the edge is the edge of the image area of ​​the virtual display. An information processing device according to any one of configurations 1 to 12, characterized by the above.

[0081] [Composition 14] A means for obtaining the position of the user's hand, A control means for controlling the placement of image content in a virtual space, The system includes a determination means for determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination means determines that the hand has interfered with the subject if the hand interferes with the inside of a predetermined surface relating to the image content and then interferes with the subject. An information processing device characterized by the following:

[0082] [method] A step to obtain the user's hand position, A control step that controls the placement of image content in a virtual space, The system includes a determination step of determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, In the determination step, if the hand first interferes with a predetermined surface relating to the image content when it begins to interfere with the image content, then even if the hand interferes with the subject, it is not determined that the hand has interfered with the subject. An information processing device characterized by the following:

[0083] [program] A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 14.

[0084] [system] A device for acquiring the position of the user's hand, A control device that controls the placement of image content in a virtual space, The device includes a determination device that determines interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination device will not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject. An information processing system characterized by the following:

Claims

1. A means for obtaining the position of the user's hand, A control means for controlling the placement of image content in a virtual space, The system includes a determination means for determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination means will not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject. An information processing device characterized by the following:

2. The determination means does not determine whether the hand and the subject have interfered if the hand first interferes with the edge when it begins to interfere with the image content. The information processing apparatus according to feature 1.

3. The determination means determines whether the hand has interfered with the subject if, when the hand begins to interfere with the image content, it does not initially interfere with the edge of the image content. The information processing apparatus according to feature 1.

4. The determination means determines that interference between the hand and the subject has occurred if the hand first interferes with the upper or lower edge when it begins to interfere with the image content, but does not determine that the hand has interfered with the subject, even if the hand interferes with the subject, if the hand first interferes with the right or left edge when it begins to interfere with the image content. The information processing apparatus according to feature 1.

5. The determination means determines that, when the distance between the user and the image content is less than a threshold, if the hand first interferes with the upper or lower edge when it begins to interfere with the image content, then interference between the hand and the subject is determined; however, if the hand first interferes with the right or left edge when it begins to interfere with the image content, then even if the hand interferes with the subject, the determination means does not determine that the hand has interfered with the subject. The information processing apparatus according to feature 1.

6. The determination means further comprises a generation means for generating haptic feedback when it is determined that the hand and the subject have interfered with each other. The information processing apparatus according to feature 1.

7. If the determination means determines that the hand and the subject have interfered with each other, the display of the subject is changed. The information processing apparatus according to feature 1.

8. The subject is a subject located behind the image content. The information processing apparatus according to feature 1.

9. The acquisition means acquires the position of the hand based on the captured image. The information processing apparatus according to feature 1.

10. The aforementioned image content is an image content that reproduces the depth relationship of the subject at the time of shooting. The information processing apparatus according to feature 1.

11. The control means controls the transparency of the image content to increase or hide the image content if the hand first interferes with the edge of a predetermined surface relating to the image content when it begins to interfere with the image content. The information processing apparatus according to feature 1.

12. The control means maintains the transparency of the image content if, when the hand begins to interfere with the image content, it first interferes with the inside of a predetermined surface relating to the image content. The information processing apparatus according to feature 11.

13. The predetermined surface is a virtual display that displays the image content, and the edge is the edge of the image area of ​​the virtual display. The information processing apparatus according to feature 1.

14. A means for obtaining the position of the user's hand, A control means for controlling the placement of image content in a virtual space, The system includes a determination means for determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination means determines that the hand has interfered with the subject if the hand interferes with the inside of a predetermined surface relating to the image content and then interferes with the subject. An information processing device characterized by the following:

15. A step to obtain the user's hand position, A control step that controls the placement of image content in a virtual space, The system includes a determination step of determining interference between the hand in the virtual space and the subject of the image content based on the position of the hand, In the determination step, if the hand first interferes with a predetermined surface relating to the image content when it begins to interfere with the image content, then even if the hand interferes with the subject, it is not determined that the hand has interfered with the subject. An information processing device characterized by the following:

16. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

17. A device for acquiring the position of the user's hand, A control device that controls the placement of image content in a virtual space, The device includes a determination device that determines interference between the hand in the virtual space and the subject of the image content based on the position of the hand, The determination device will not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject. An information processing system characterized by the following:

Images