Information processing device, information processing device control method, display device, display device control method, system and program

Abstract

To provide a technology for improving user-friendliness by attaching a controller to a desired site, and by allocating the function of the controller in accordance with a predetermined site.SOLUTION: An information processing device includes: obtaining means for obtaining a picked-up image from an imaging unit; determining means for determining, from the picked-up image obtained by the obtaining means, a position where a controller is attached or is held; and control means for executing, based on the determination result by the determining means, a control so as to decide a process to be executed in accordance with an input from the controller.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly fuses the real world and the virtual world in real time. One MR technology is an MR system that uses a video see-through type HMD (Head Mounted Display: hereinafter referred to as "HMD"). In the MR system, an image of a subject observed from the pupil position of a user wearing the HMD is captured by an image capture unit built into the HMD, and an image in which CG (Computer Graphics) is superimposed on the captured image is presented to the user, allowing the user to experience an MR space.

[0003] In the MR system, a controller is used for a user wearing an HMD to operate CG in the virtual world and a UI (User Interface). Based on the position and orientation information of the controller and the operation information of the cursor, it is possible to move a virtual light ray, called a ray, or a virtual pointer displayed on the HMD, and to select or determine CG or a setting menu based on button operation information. These controllers are used by the user by holding them in the hand or wearing them on the fingers, and many of them can be used with either the left or right hand. For example, the following Patent Document 1 discloses a technology for changing the assignment of each button in a ring-type controller worn on the thumb depending on whether the controller is worn on the thumb of the left or right hand. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-077302 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology disclosed in Patent Document 1, the allocation of each button is changed depending on whether the pointing device is worn on the thumb of the left or right hand. However, it is not assumed that the function of the controller itself will change when the controller is worn on a specific finger other than the thumb. The pointing device disclosed in Patent Document 1 cannot be worn on a finger other than the thumb, and depending on the situation in which the MR system is operated, convenience may be impaired. Therefore, the present invention aims to improve convenience by wearing the controller on a desired part and allocating the controller's functions according to a specific part. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device characterized by having an acquisition means for acquiring an image from an imaging unit, a determination means for determining a position at which a controller is worn or held from the image acquired by the acquisition means, and a control means for controlling to determine a process to be executed in response to an input from the controller based on a determination result of the determination means. Effect of the Invention

[0007] According to the present invention, it is possible to improve convenience by attaching a controller to a desired part and allocating functions of the controller according to a predetermined part. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram for explaining a first embodiment. [Diagram 3] 5A to 5C are diagrams illustrating a process of generating a composite image from a captured image and an image in a virtual space according to the first embodiment. [Figure 4]FIG. 1 is a diagram illustrating an example in which a virtual object and a UI are operated in an MR space using two controllers. [Diagram 5] FIG. 13 is a diagram illustrating an example in which a virtual object is operated in an MR space using three controllers. [Figure 6] 5A to 5C are diagrams illustrating the process of recognizing the attachment position of the controller according to the first embodiment. [Figure 7] 4 is a flowchart of a process for recognizing the attachment position of a controller according to the first embodiment. [Figure 8] 13A to 13C are diagrams illustrating an image of a mixed reality space displayed on a display unit of an HMD according to a second embodiment. [Figure 9] 13A to 13C are diagrams illustrating a process for recognizing the attachment position of a controller according to a second embodiment. [Figure 10] 1A is a block diagram showing an example of the hardware configuration of an HMD 100; FIG. 1B is a block diagram showing an example of the hardware configuration of a computer device; and FIG. 1C is a block diagram showing an example of the hardware configuration of a controller 120. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (First embodiment) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by a wired connection such as a USB cable, or wirelessly by a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that, although a PC will be described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. As shown in FIG. 1, the controller 120 may be in the shape of a finger ring that can be worn and supported by the user, or in the shape of a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing a decision operation or a selection operation on the display. The controller 120 performs wireless communication with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, and may be a system that communicates with the HMD 100.

[0014] By moving the controller 120, the user can change the indicated position on the display according to the movement of the controller. The indicated position may be represented by a point, or may be represented by a virtual ray (ray) connecting the point of the indicated position and the controller with a straight line (line segment) or a dotted line. By pressing a physical button, a menu determination operation or a selection operation can be performed. Although the shape of the controller 120 is assumed to be a ring type or a hand-held type, it is not limited thereto as long as it can be supported by a finger, a hand, or an arm. Also, although the button is assumed to be a physical button, it may be operable like a track pad, a touch panel, a wheel, or a track ball, and in addition to pressing the button, a slide operation, a flick operation, or a touch operation may be used.

[0015] Note that the controller may be attached to at least one of a finger, a hand, or an arm.

[0016] Note that the controller may be attached to an object held by a hand, and the position information and attitude information of the attached position may be acquired from the sensor and used. Examples of such an object include an object imitating a tool.

[0017] <Internal Configuration of HMD> Referring to FIG. 2, the internal configuration of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, and a working memory 206.

[0018] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. In other words, the imaging unit 202 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the HMD 101. By measuring distances with the stereo cameras, information on the distances from the two cameras to the subject can be acquired as distance information. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight direction of the person wearing the HMD.

[0020] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. Images of a subject (a range in front of the user) captured by two cameras are output to the PC 110 as captured images. The captured images are output to the control unit 201. The imaging unit 202 may also capture and output video.

[0021] The image display unit 203 displays the composite image. The image display unit 203 has a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, an organic EL panel is disposed in front of each eye of the user. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is superimposed directly on the real space seen through the half mirror. Also, the image display unit 203 may display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0022] The attitude sensor unit 204 acquires attitude (and position) information of the HMD 100. Then, the attitude sensor unit 204 acquires attitude information of the user (user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the information on the user's attitude (attitude information) to the PC 110. Note that the attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0023] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include points that are characteristic of parts such as the joints and fingertips of the fingers, the back of the hand (palm), and the arms. Each joint point indicates a coordinate position, and the posture can be estimated from information on a plurality of joint points. A known object recognition or pose estimation method of machine learning using a convolutional neural network can be used as a method for estimating the position or posture of the hand and each joint point of the hand. In addition, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation using stereo matching using the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0024] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 211, which will be described later.

[0025] The volatile memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0026] The gaze imaging unit 207 is a camera that acquires an image for detecting the gaze of the user, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0027] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0029] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have multiple input formats.

[0030] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. If there are multiple controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0031] The controller attitude sensor unit 224 includes an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0032] The output unit 225 is composed of an LED light source, a speaker, a vibration element, etc.

[0033] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a volatile memory 213, a communication unit 214, and a recording medium 215.

[0034] The control unit 211 is a CPU that controls each unit of the PC 110 according to input signals and a program described later. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by having a plurality of hardware share the processing. The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0035] The control unit 211 also obtains the number of controllers included in the captured image. It also executes a process for recognizing the attachment position of each controller, using the information obtained via the communication unit 214. Then, depending on the recognition result, it performs control to change the operation content for the input information of each controller for each controller.

[0036] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0037] In addition, the control unit 211 receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the volatile memory 213.

[0038] Furthermore, in the control unit 211, the communication unit 214 receives information on a change in the position or attitude of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image an indication position according to the information on the change in the position or attitude of the controller 120. Note that the control unit 211 may also superimpose on the combined image an indication position according to the information on the change in the position and attitude of the controller 120.

[0039] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate a composite image).

[0040] The volatile memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0041] The wrist joints may be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. Note that the control unit 211 may estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0042] In addition to the above, the MR system also includes other components, such as a control unit of either the HMD 100 or the PC 110, which performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, this is not the focus of the present invention, so a description thereof will be omitted.

[0043] <Process for generating composite image> With reference to FIG. 3, a process of generating a composite image from a captured image and an image of a virtual space will be described. A captured image 301 includes a marker 302 artificially arranged in a real space. In FIG. 3, the number of markers is one for simplicity of explanation, but in reality, a plurality of markers may be included. In addition, when performing position and orientation estimation by SLAM (Simultaneous Localization and Mapping) using natural feature points in an image, it is not necessary to arrange a marker. The PC 110 extracts the marker 302 and natural feature points included in the captured image 301, and obtains the position and orientation of the HMD 100 based on the extracted marker 302 and natural feature points and the position and orientation received from the control device 102. Then, the PC 110 generates an image 303 of the virtual space seen from a viewpoint having the obtained position and orientation. The image 303 includes a virtual object 304. The PC 110 generates an image 305 of a mixed reality space, which is a composite image obtained by combining the captured image 301 and the image 303 of the virtual space, and transmits the generated image 305 to the HMD 100. When combining a captured image with an image in a virtual space, information regarding the depth in three-dimensional space and information regarding the transparency of virtual objects can be used to generate a composite image that takes into account the front-to-back relationship between real objects and virtual objects, or a composite image in which virtual objects are combined in a semi-transparent state.

[0044] <Example of operation using multiple controllers> Next, operations using multiple controllers according to the present embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram illustrating an example in which a virtual object and a UI are operated in an MR space using two controllers.

[0045] A user wearing the HMD wears controller 120a on the index finger of his left hand and controller 120b on the index finger of his right hand. The user operates a ray 401 based on input information from a controller orientation sensor unit 224a of controller 120a worn on the index finger of his left hand, and selects a virtual object 304 pointed to by the ray by operating a button. Meanwhile, an operation menu 402 is displayed by operating a button on controller 120b worn on the index finger of his right hand, a menu is selected from the operation menu based on input information from controller 120b, and the selected menu is confirmed by operating a button.

[0046] FIG. 5 is a diagram illustrating an example in which a virtual object is operated in an MR space using three controllers.

[0047] A user wearing an HMD wears a controller 120a on the index finger of his left hand, a controller 120b on the index finger of his right hand, and a controller 120c on the thumb of his right hand. The user operates a ray 401 based on input information from a controller orientation sensor unit 224a of the controller 120a worn on the index finger of his left hand, and selects a virtual object 304 pointed to by the ray by operating a button. Meanwhile, the virtual object 304 is enlarged or reduced based on input information from a controller orientation sensor unit 224b of the controller 120b worn on the index finger of his right hand and a controller orientation sensor unit 224c of the controller 120c worn on the thumb. For example, assume that the user moves the thumb and index finger away from each other with the right hand, that is, performs a pinch-out motion, as shown by the arrows in FIG. 5. In this case, based on the input information from the controller attitude sensor unit 224b and the controller attitude sensor unit 224c, the virtual object 304 selected on the controller 120a worn on the left hand is enlarged as indicated by the arrow, and the display is controlled so that it becomes virtual object 504.

[0048] 4, the operation menu may be displayed by operating the buttons on controller 120b worn on the index finger of the right hand. Other operations may be assigned, such as displaying a setting menu for the imaging unit and display unit of the HMD by operating the buttons on controller 120c worn on the thumb of the right hand.

[0049] In this way, even with the same type of controllers, by varying the processing of input information by the attitude sensor unit and the operation unit depending on the number of controllers being used and the fingers on which they are worn, it becomes possible to perform complex and flexible operations in the MR space. Note that the processing of input information by the attitude sensor unit and the operation unit may be assigned depending on not only the number of controllers and the fingers on which they are worn, but also the combination of the fingers on which they are worn.

[0050] <Controller installation position determination process> The process of determining the attachment position of the controller in the MR system according to this embodiment will be described with reference to FIG.

[0051] In an MR system, in order to perform operations using multiple controllers as described with reference to Figures 4 and 5, it is necessary to determine the number of controllers to be used simultaneously and the attachment positions of each controller. In the MR system of this embodiment, the number of controllers to be used simultaneously and the attachment positions of each controller can be determined by using captured images acquired by the imaging unit of the HMD.

[0052] The control unit 211 of the PC 110 controls the image display unit 203 of the HMD 100 to synthesize the captured image acquired by the imaging unit 202 of the HMD 100 with the virtual hand-shaped guides 603L and 603R, generate a composite image 605, and display the composite image 605 on the image display unit 203 of the HMD 100. That is, the control unit 211 of the PC 110 superimposes the hand-shaped guide on the captured image to generate a superimposed image. The user places his left hand on the guide 603L and his right hand on the guide 603R. The control unit 211 of the PC 110 performs SLAM processing using the captured image acquired when the user places both hands on the guides 603L and 603R, and detects the controllers 120a, 120b, and 120c. When the shape of the controller 120 to be used is known to the control unit 211 of the PC 110, the control unit 211 of the PC 110 searches for a group of natural feature points that match the shape of the controller 120 from among the natural feature points extracted from the captured image, and detects the controller 120. This process detects the number of controllers attached to the user's fingers that are included in the captured image. In order to clearly indicate the detected controllers, the control unit 211 of the PC 110 may draw the controller detection results 604a, 604b, and 604c and display them by combining them with the captured image. The user may place his or her hand in accordance with the guide, or the guide may be displayed in accordance with the hand recognized in the captured image. In FIG. 6, the back of the hand is shown to the imaging unit, but the palm may be shown, and the hand may not be spread out, and any pose may be used as long as the hand can be recognized.

[0053] Next, the control unit 211 of the PC 110 transmits a control signal for illuminating the LED of the output unit 225 of the controller 120. The control unit 211 of the PC 110 can grasp the identification information of each controller from the connection information at the time of wireless connection between the controller and the HMD, so that it is possible to control each controller individually. The control unit 211 of the PC 110 transmits a different LED light emission control signal to each controller, and detects the light emission state of each controller's LED from the captured image, thereby recognizing the correspondence between the identification information of the controller and the attachment position, i.e., the finger on which the controller is worn. The light emission control method includes a method of sequentially illuminating the LED of each controller with a time difference, a method of varying the light emission pattern such as the blinking interval, and a method of varying the light emission color. Note that, although the light emission of the LED is given as an example of the output unit 225 here, the controller may be given a vibration function and controlled to vibrate. The control unit 211 of the PC 110 transmits a control signal for vibrating the vibration device to each controller with a time difference, and the user performs an action such as bending the finger on which the vibrating controller is worn. The control unit 211 of the PC 110 can recognize the correspondence between the identification information of the controller and the attachment position, i.e., the finger on which the controller is worn, by detecting the movement of the user's finger from the captured image. Note that the controller may be provided with a sound output function and controlled to emit sounds from the controller.

[0054] Note that the position where the controller is worn may be determined based on the detected position of the controller and the positional relationship with the handprint.

[0055] The positions of the hands and the controller may be recognized by a hand masking process. That is, a color range including the color of the user's hand may be extracted from the captured image, the region of the hand may be recognized in the captured image, and the controller may be recognized as being worn in a region of the hand that is not included in the color range including the color of the user's hand.

[0056] In this way, in the MR system of this embodiment, the captured images acquired by the imaging section of the HMD 100 are used to recognize the number of controllers used simultaneously and the attachment positions of each controller.

[0057] Furthermore, in the process of determining the attachment position of the controller described above, if the number of controllers connected to the HMD 100 is greater than the number of controllers detected from the captured image, it can be determined that the controller is not used by the user. That is, although the same controller exists within the range of wireless connection with the HMD, it can be determined that the controller is not used by the user. By disconnecting the wireless connection with such a controller, that is, by stopping communication, it is possible to reduce the power consumption of the HMD and the battery consumption of the unused controller.

[0058] <Flowchart relating to determination process> A flowchart of a process for determining the attachment position of a controller in the MR system according to this embodiment will be described with reference to FIG.

[0059] In step S701, the control unit 211 starts acquiring a captured image of the real space by the imaging unit 202 of the HMD.

[0060] In step S702, the control unit 211 performs processing for drawing the guides 603L and 603R and synthesizing and displaying them on the captured image.

[0061] In step S703, the control unit 211 detects the number of controllers worn by the user through SLAM processing, using a captured image in which the user places both hands in accordance with the displayed guide.

[0062] In step S704, the control unit 211 acquires connection information of the controller that is wirelessly connected to the HMD.

[0063] In step S705, the control unit 211 controls the output unit 225 of each controller based on the connection information of the controller.

[0064] In step S706, the control unit 211 detects the output result from the output unit 225 of each controller, or the user's response to the output, from the captured image, and determines the attachment position of each controller.

[0065] In step S707, the control unit 211 determines whether or not the determination of the attachment positions of all controllers wirelessly connected to the HMD 100 has been completed. If the control unit 211 determines that the determination of the attachment positions of all controllers has been completed, the process proceeds to step S709. If the control unit 211 does not determine that the determination of the attachment positions of all controllers has been completed, the process proceeds to step S708.

[0066] In step S708, the control unit 211 performs processing to release the wireless connection between the controller whose attachment position cannot be recognized and the HMD, that is, processing to stop communication, and proceeds to step S709.

[0067] In step S709, the control unit 211 performs control to determine the process to be executed in response to the input from the controller based on the determination result of the attachment positions of the controllers. That is, the control unit 211 performs control to change the operation content for the input information from each controller in response to the combination of the determined number of controllers and the attachment positions.

[0068] In the present embodiment, the captured image acquired by the imaging unit 202 of the HMD 100 is used as the captured image used when recognizing the controller, but the means for acquiring the captured image is not limited to this. For example, the captured image may be acquired by an objective camera such as a stereo camera device included in the PC 110 and connected as a peripheral device of the PC 110. Similarly, the image display unit 203 of the HMD 100 is used to display to the user the captured image on which the guide 603 used when recognizing the controller is composited, but this may also be achieved by using, for example, a monitor connected as a peripheral device of the PC 110.

[0069] In addition, the controller 120 has been described as being worn on the user's finger when used, but this is not limiting, and the controller 120 may be worn on a predetermined part of the user. For example, the controller may be a controller that the user can hold in his / her hand and use, or a watch-type controller worn on the arm. Alternatively, the controller may be a controller that can be attached to a tool held in the hand and allows the user to operate the input means while attached to the tool. In a controller that is held in the hand and used, the number of controllers and which hand is using them can be recognized by a method of displaying a guide in a captured image. In addition, in a controller that is attached to a tool held in the hand and used, if the shape of the tool is known to the control unit 211 of the PC 110, the number of controllers and the attachment position can be recognized by a method of displaying a guide in a captured image.

[0070] As described above, in the MR system of this embodiment, the operation contents for the input information from the controllers are changed according to the number of controllers used simultaneously by the user and their mounting positions, allowing the user to perform more complex and flexible operations. Furthermore, in the MR system of this embodiment, it is also possible to use different controllers simultaneously. In that case, the operation contents for the input information from each controller are changed according to the type and number of detected controllers and the mounting position (wearing position) of each controller.

[0071] Second Embodiment In each of the following embodiments including this embodiment, the difference from the first embodiment will be described, and it will be assumed that the embodiments are the same as the first embodiment unless otherwise specified below. In the first embodiment, a configuration was described in which the result of the SLAM process executed by the control unit 211 is used to recognize the number of controllers used by the user and the attachment positions of each controller. In this embodiment, a configuration is described in which the result of the hand detection process used in the MR system is used to recognize the number of controllers used by the user and the attachment positions of each controller.

[0072] <Occlusion Technology> The hand detection process used in the MR system according to this embodiment will be described with reference to FIG.

[0073] 8(a) and 8(b) are diagrams for explaining an image of a mixed reality space displayed on the image display unit 203 of the HMD 100 in the MR system. In order for a user wearing the HMD to feel as if a virtual object 304 actually exists in the image 305 of the mixed reality space, it is necessary to draw the CG of the virtual object 304 while taking into consideration the front-to-back relationship between the user's hand 801 and the virtual object 304. That is, when the hand 801 is in front of the virtual object 304 as seen from the HMD 100, it is necessary to display the hand 801 hiding the virtual object 304 as shown in FIG. 8(a). On the other hand, when the hand 801 is behind the virtual object 304, it is necessary to display the hand 801 hiding the virtual object 304 as shown in FIG. 8(b). In the MR system of this embodiment, in order to perform such a display, the control unit 211 of the PC 110 extracts the area of ​​the hand 801 from the captured image acquired by the imaging unit of the HMD 100, and determines the depth of the CG of the virtual object 304 according to the information on the depth. The hand 801 region is extracted by detecting pixels included in a color region that has been registered in advance as the color of the hand from the captured image and extracting the region as the hand. Furthermore, by generating a bone model of the detected hand region, the posture of the hand can be estimated and the depth accuracy can be improved.

[0074] <Controller installation position determination process according to the second embodiment> The process of recognizing the attachment position of the controller in the MR system according to this embodiment will be described with reference to FIG.

[0075] In the MR system of this embodiment, as in the first embodiment, the number of controllers being used simultaneously and the attachment position of each controller can be recognized by utilizing the captured images obtained by the imaging unit of the HMD.

[0076] In the control unit 211 of the PC 110, SLAM processing is performed using the captured image acquired by the imaging unit of the HMD, and the controllers 120a and 120b are detected. Note that, in order to clearly indicate the detected controllers, the control unit 211 of the PC 110 may draw frames 902a and 902b indicating the detection results of the controllers and display them by combining them with the captured image.

[0077] Furthermore, the control unit 211 detects the hand region from the captured image by image recognition, and generates bone models 903L and 903R of the detected hand region.

[0078] Based on the controller detection results and the generated bone model, the control unit 211 can associate the number of detected controllers with which fingers of the user are wearing the controllers.

[0079] Next, the control unit 211 determines the correspondence between the identification information of the controller and the finger on which the controller is worn, based on the correlation between the position and posture of the hand estimated using the bone model and the position and posture from each controller. For example, when at least one of the position and posture of the right hand changes while the left hand is stationary, the inertial information from the controller worn on the index finger of the right hand changes, and the controller is determined to be worn on the index finger of the right hand by comparing it with the captured image. The same process is performed for all controllers detected by the control unit 211 until the fingers on which the controller is worn can be recognized.

[0080] Also, as shown in Fig. 9, when one controller is worn on each hand, or when one controller is worn on either the left or right hand, it is possible to more easily recognize the attachment position. That is, in the case of controllers worn in opposite directions on the left and right hands, such as controller 120 in this embodiment, the attachment position can be determined even if the hands are kept still in the state shown in Fig. 9 during the recognition process. Control unit 211 can detect the orientation of the controller from the orientation information of the controller itself indicated by the orientation sensor of the controller, or information on the geomagnetic direction indicated by a geomagnetic sensor built into the orientation sensor, and can recognize the attachment position from that orientation.

[0081] In this way, in the MR system of this embodiment, by utilizing the results of the hand detection processing used in the MR system, it is possible to recognize the number of controllers being used and the attachment positions of each controller, as in the first embodiment.

[0082] Furthermore, the process for identifying the number of controllers used and the attachment positions of each controller in the control unit 211 can be realized using the same process as in the first embodiment. Furthermore, if one recognition process cannot identify the attachment positions of all the controllers, the process may be switched to another recognition process and executed sequentially.

[0083] (Third embodiment) Each functional unit in the HMD 100, the PC 110, and the controller 120 shown in FIG. 2 may be implemented by hardware, or some of the functional units may be implemented by software (computer program).

[0084] In the latter case, the imaging unit 202, the orientation sensor unit 204, and the image display unit 203 may be implemented in hardware, and the remaining functional units may be implemented in software in the HMD 100. In this case, the software is stored in a memory of the HMD 100, and the processor of the HMD 100 executes the software to realize the functions of the corresponding functional units.

[0085] An example of the hardware configuration of such an HMD 100 will be described with reference to the block diagram in Fig. 10(a). The processor 1010 executes various processes using computer programs and data stored in the RAM 1020. As a result, the processor 1010 controls the operation of the entire HMD 100, and executes or controls each of the processes described above as being performed by the HMD 100.

[0086] The RAM 1020 has an area for storing computer programs and data loaded from the non-volatile memory 1030, and an area for storing data received from the PC 110 and the controller 120 via an I / F (interface) 1070. The RAM 1020 also has a work area used when the processor 1010 executes various processes. In this way, the RAM 1020 can provide various areas as needed.

[0087] The non-volatile memory 1030 stores computer programs and data for causing the processor 1010 to execute or control the above-mentioned operations of the HMD 100. The computer programs stored in the non-volatile memory 1030 include computer programs for causing the CPU 1001 to execute the functions of the functional units (excluding the imaging unit 202, the attitude sensor unit 204, and the image display unit 203) of the HMD 100 shown in Fig. 2. The computer programs and data stored in the non-volatile memory 1030 are loaded into the RAM 1020 as appropriate under the control of the processor 1010, and become targets for processing by the processor 1010.

[0088] The imaging unit 1040 includes the imaging unit 202 described above. The orientation sensor 1050 includes the orientation sensor unit 204 described above. The display unit 1060 includes the image display unit 203 described above. The I / F 1070 includes a communication unit for connecting to the PC 110. The processor 1010, the RAM 1020, the non-volatile memory 1030, the imaging unit 1040, the orientation sensor 1050, the display unit 1060, and the I / F 1070 are all connected to a bus 1090. Note that the configuration shown in FIG. 10(a) is an example of a configuration applicable to the HMD 100, and can be changed / modified as appropriate.

[0089] In addition, the PC 110 can be applied to any computer device that can execute software corresponding to each functional unit except for the communication unit 214. An example of the hardware configuration of a computer device that can be applied to the PC 110 will be described with reference to the block diagram of FIG.

[0090] The CPU 1001 executes various processes using computer programs and data stored in the RAM 1002 and the ROM 1003. As a result, the CPU 1001 controls the operation of the entire computer device, and executes or controls each of the processes described above as being performed by the PC 110 to which the computer device is applied.

[0091] The RAM 1002 has an area for storing computer programs and data loaded from the ROM 1003 or the external storage device 1006, and an area for storing data received from the HMD 100 via the I / F 1007. The RAM 1002 also has a work area used when the CPU 1001 executes various processes. In this way, the RAM 1002 can provide various areas as appropriate. The ROM 1003 stores setting data, startup programs, and the like for the computer device.

[0092] The operation unit 1004 is a user interface such as a keyboard, a mouse, a touch panel, etc., and the user can input various instructions to the CPU 1001 by operating it.

[0093] The display unit 1005 is configured with a liquid crystal screen, a touch panel screen, or the like, and can display the processing results by the CPU 1001 as images, characters, etc. The display unit 1005 may be a projection device such as a projector that projects images and characters.

[0094] The external storage device 1006 is a large-capacity information storage device such as a hard disk drive device. An OS (operating system) is stored in the external storage device 1006. The external storage device 1006 also stores computer programs and data for causing the CPU 1001 to execute the functions of each functional unit (except for the communication unit 214) of the PC 110 shown in FIG.

[0095] Computer programs and data stored in the external storage device 1006 are loaded into the RAM 1002 as appropriate under the control of the CPU 1001 and are processed by the CPU 1001 .

[0096] The I / F 1007 is a communication interface for performing data communication with the controller 120, and functions as the above-mentioned communication unit 214. In other words, this computer device performs data communication with the controller via the I / F 1007.

[0097] The CPU 1001, RAM 1002, ROM 1003, operation unit 1004, display unit 1005, external storage device 1006, and I / F 1007 are all connected to a bus 1008. Note that the configuration shown in Fig. 10(b) is an example of a configuration applicable to the PC 110, and can be changed / modified as appropriate.

[0098] In the controller 120, the operation unit 222, communication unit 223, controller attitude sensor unit 224, and output unit 225 may be implemented in hardware, and the remaining functional units may be implemented in software. In this case, the software is stored in a memory of the controller 120, and the processor of the controller 120 executes the software to realize the functions of the corresponding functional units.

[0099] An example of the hardware configuration of such a controller 120 will be described with reference to the block diagram of Fig. 10(c). The processor 1100 executes various processes using computer programs and data stored in the RAM 1200. As a result, the processor 1100 controls the operation of the entire controller 120, and executes or controls each of the processes described above as being performed by the controller 120.

[0100] The RAM 1200 has an area for storing computer programs and data loaded from the non-volatile memory 1300, and an area for storing data received from the HMD 100 via the I / F 1700. The RAM 1200 also has a work area used when the processor 1100 executes various processes. In this way, the RAM 1200 can provide various areas as needed.

[0101] The non-volatile memory 1300 stores computer programs and data for causing the processor 1100 to execute or control the operation of the controller 120. The computer programs stored in the non-volatile memory 1300 include a computer program for causing the CPU 1001 to execute the functions of the functional units of the controller 120 shown in Fig. 2. Here, the functional units of the controller 120 are functional units other than the operation unit 222, the communication unit 223, the controller attitude sensor unit 224, and the output unit 225. The computer programs and data stored in the non-volatile memory 1300 are loaded into the RAM 1200 as appropriate under the control of the processor 1100, and become targets for processing by the processor 1100.

[0102] The input unit 1400 includes the above-mentioned operation unit 222. The attitude sensor 1500 includes the above-mentioned controller attitude sensor unit 224. The output unit 1600 includes the above-mentioned output unit 225. The I / F 1700 includes the above-mentioned communication unit 223. The processor 1100, RAM 1200, non-volatile memory 1300, input unit 1400, attitude sensor 1500, output unit 1600, and I / F 1700 are all connected to a bus 1800. Note that the configuration shown in Fig. 10(c) is an example of a configuration applicable to the controller 120, and can be changed / modified as appropriate.

[0103] (Fourth embodiment) The configuration of the MR system shown in Fig. 3 is an example, and each process described above as being performed by the HMD 100 may be shared and executed by multiple devices. Also, each process described above as being performed by the PC 110 may be shared and executed by multiple devices. Also, each process described above as being performed by the controller 120 may be shared and executed by multiple devices.

[0104] Moreover, instead of the head-mounted display device, a "portable device having an imaging unit, an orientation sensor, and a display unit" such as a smartphone may be used. Moreover, such a portable device may be added to the MR system in addition to the head-mounted display device. In such a case, the PC 110 generates an image of the mixed reality space according to the position and orientation of the head-mounted display device and delivers it to the head-mounted display device, and generates an image of the mixed reality space according to the position and orientation of the portable device and delivers it to the portable device. The method of generating the image of the mixed reality space is the same as in the above embodiment.

[0105] Furthermore, the HMD 100 and the PC 110 may be integrated together, or instead of the head-mounted display device, the above-mentioned portable device and the PC 110 may be integrated together.

[0106] In addition, in the above embodiment, the posture sensor unit 204 has been described as being included in the HMD 100, but this is not limited thereto. For example, the posture sensor unit 204 may be configured to obtain necessary information from an image captured by an objective camera installed around the wearer of the HMD 100.

[0107] In addition, the numerical values, processing execution timing, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and it is not intended that each embodiment be limited to these examples.

[0108] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0109] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0110] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0111] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0112] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0113] [Configuration 1] An acquisition means for acquiring a captured image from an imaging unit; a determination means for determining a position at which a controller is worn or held from the captured image acquired by the acquisition means; a control means for controlling the determination of a process to be executed in response to an input from the controller based on a determination result of the determination means; 13. An information processing device comprising:

[0114] [Configuration 2] The present invention further includes a first detection means for detecting a predetermined part of a user from the captured image acquired by the acquisition means, the determination means determines a predetermined part of the user wearing or holding the controller based on a detection result of the first detection means. 2. The information processing device according to configuration 1.

[0115] [Configuration 3] a second detection means for detecting a controller from the captured image acquired by the acquisition means; 3. The information processing device according to configuration 1 or 2.

[0116] [Configuration 4] the determination means determines a predetermined part of the user wearing or holding the controller based on a detection result of the first detection means and a detection result of the second detection means. 4. The information processing device according to configuration 3.

[0117] [Configuration 5] A communication means for communicating with the controller; and second control means for controlling the controller detected by the second detection means to output light or vibration when the controller is connected via the communication means. 5. The information processing device according to configuration 3 or 4.

[0118] [Configuration 6] when the second detection means detects the light or the vibration output by the controller, the determination means determines a position at which the controller is being worn or held. 6. The information processing device according to any one of configurations 3 to 5.

[0119] [Configuration 7] when the second detection means detects a controller that is not worn or held in a predetermined part of the user among the controllers detected from the captured image, the control means stops communication with the controller that is not worn or held in a predetermined part of the user. 7. The information processing device according to any one of configurations 3 to 6.

[0120] [Configuration 8] When the second detection means does not detect the light or the vibration output from the controller among the controllers detected from the captured image, the control means performs control to stop communication with the controller that does not detect the light or the vibration. 8. The information processing device according to any one of configurations 3 to 7.

[0121] [Configuration 9] Further, a third detection means for detecting an external device to which the controller can be attached is provided. 9. The information processing device according to any one of configurations 1 to 8.

[0122] [Configuration 10] the determination means determines a position of the controller attached to the external device from the captured image acquired by the acquisition means. 10. The information processing device according to configuration 9.

[0123] [Configuration 11] the first detection means detects a first region having pixels of a color included in a color range including the color of the predetermined part of the user; 3. The information processing device according to configuration 2.

[0124] [Configuration 12] The first detection means detects a finger of the user, the determination means determines the finger wearing the controller based on a detection result of the first detection means. 12. The information processing device according to configuration 2 or 11.

[0125] [Configuration 13] The first detection means detects a hand of the user, the determination means determines a hand holding the controller based on a detection result of the first detection means. 12. The information processing device according to configuration 2 or 11.

[0126] [Configuration 14] The controller is a ring controller. 14. The information processing device according to any one of configurations 1 to 13.

[0127] [Configuration 15] The method further includes a generating unit for generating a superimposed image by superimposing a handprint on the captured image acquired by the acquiring unit, the determining means determines a position where the controller is attached from the superimposed image. 15. The information processing device according to any one of configurations 1 to 14.

[0128] [Configuration 16] the determination means determines a position where the controller is worn based on a positional relationship between the detected position of the controller and the handprint. 16. The information processing device according to configuration 15.

[0129] [Configuration 17] a second acquisition means for acquiring information about an attitude of the controller from the controller, the determination means determines a position where the controller is worn or held, based on the captured image acquired by the acquisition means and information regarding the attitude of the controller acquired by the second acquisition means. 17. The information processing device according to any one of configurations 1 to 16.

[0130] [Control method] An acquisition step of acquiring a captured image from an imaging unit; a determination step of determining a position where a controller is worn or held from the captured image acquired in the acquisition step; a control step of controlling to determine a process to be executed in response to an input from the controller based on a result of the determination step; 13. A method for controlling an information processing apparatus comprising the steps of:

[0131] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[0132] [system] A controller; an acquisition device for acquiring an image from an imaging unit; a determination device that determines a position at which a controller is worn or held from the captured image acquired by the acquisition device; a control device that determines a process to be executed in response to an input from the controller based on a determination result of the determination device; An information processing system comprising:

Claims

1. An acquisition means for acquiring a captured image from an imaging unit; a determination means for determining a position at which a controller is worn or held from the captured image acquired by the acquisition means; a control means for controlling the determination of a process to be executed in response to an input from the controller based on a determination result of the determination means; 13. An information processing device comprising:

2. The present invention further includes a first detection means for detecting a predetermined part of a user from the captured image acquired by the acquisition means, the determination means determines a predetermined part of the user wearing or holding the controller based on a detection result of the first detection means.

2. The information processing apparatus according to claim 1,

3. a second detection means for detecting a controller from the captured image acquired by the acquisition means; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the determination means determines a predetermined part of the user wearing or holding the controller based on a detection result of the first detection means and a detection result of the second detection means.

4. The information processing apparatus according to claim 3.

5. A communication means for communicating with the controller; and second control means for controlling the controller detected by the second detection means to output light or vibration when the controller is connected via the communication means.

4. The information processing apparatus according to claim 3.

6. when the second detection means detects the light or the vibration output by the controller, the determination means determines a position at which the controller is being worn or held.

6. The information processing apparatus according to claim 5,

7. when the second detection means detects, among the controllers detected from the captured image, a controller that is not worn or held in a predetermined part of the user, the control means stops communication with the controller that is not worn or held in a predetermined part of the user.

4. The information processing apparatus according to claim 3.

8. When the second detection means does not detect the light or the vibration output from the controller among the controllers detected from the captured image, the control means performs control to stop communication with the controller that does not detect the light or the vibration.

6. The information processing apparatus according to claim 5,

9. Further, a third detection means for detecting an external device to which the controller can be attached is provided.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

10. the determination means determines a position of the controller attached to the external device from the captured image acquired by the acquisition means.

10. The information processing apparatus according to claim 9,

11. the first detection means detects a first region having pixels of a color included in a color range including the color of the predetermined part of the user; 3. The information processing apparatus according to claim 2.

12. The first detection means detects a finger of the user, the determination means determines the finger wearing the controller based on a detection result of the first detection means.

12. The information processing apparatus according to claim 2 or 11.

13. The first detection means detects a hand of the user, the determination means determines a hand holding the controller based on a detection result of the first detection means.

12. The information processing apparatus according to claim 2 or 11.

14. The controller is a ring controller.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

15. The method further includes a generating unit for generating a superimposed image by superimposing a handprint on the captured image acquired by the acquiring unit, the determining means determines a position where the controller is attached from the superimposed image.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

16. the determination means determines a position where the controller is worn based on a positional relationship between the detected position of the controller and the handprint.

16. The information processing apparatus according to claim 15,

17. a second acquisition means for acquiring information about an attitude of the controller from the controller, the determination means determines a position where the controller is worn or held, based on the captured image acquired by the acquisition means and information regarding the attitude of the controller acquired by the second acquisition means.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

18. An acquisition step of acquiring a captured image from an imaging unit; a determination step of determining a position where a controller is worn or held from the captured image acquired in the acquisition step; a control step of controlling to determine a process to be executed in response to an input from the controller based on a result of the determination step; 13. A method for controlling an information processing apparatus comprising the steps of:

19. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

20. A controller; an acquisition device for acquiring an image from an imaging unit; a determination device that determines a position at which a controller is worn or held from the captured image acquired by the acquisition device; a control device that determines a process to be executed in response to an input from the controller based on a determination result of the determination device; An information processing system comprising:

Images