Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To enable a user wearing or holding a controller on his / her hand to operate the controller in a comfortable posture for a long time.SOLUTION: In an information processor having a display part 150 provided with a display screen 400 to be visually recognized by a user, real space coordinates 421 to be the home position of a controller 200 are set on the basis of position information and operation information transmitted from the controller 200 mounted or held on the hand of the user, and the set real space coordinates 421 are converted into virtual space coordinates 402 within the range of the display screen 400.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, cross reality (XR) systems have been proposed that allow users to experience virtual reality using a head-mounted display (HMD), a type of information processing device. Many of these systems involve a controller worn or held in the hand, separate from the HMD, and the controller communicates with the HMD to transmit user operation information to the HMD and control its operation. In such systems, the controller's positional and inertial information must be accurately transmitted to the HMD in order to accurately convey the user's operational intentions. Various methods for calibrating the controller have been devised. For example, Patent Document 1 describes a method for automatically calibrating an HMD when it is placed on a designated mounting surface while not in use. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-76328 Summary of the Invention [Problem to be solved by the invention]

[0004] In this way, methods have been proposed to accurately reproduce the positional and inertial information of a real user's hands and controller in the virtual space of an HMD.

[0005] However, as XR systems become more widespread in the future, it is expected that there will be cases where it is not always desirable to accurately reproduce the positions of real hands and controllers in virtual space. For example, there may be cases where a user wearing or holding a controller in their hand must raise their hand to operate the controller for a long period of time, or where the user must raise their hand while lying down. In other words, as XR systems become more widespread in the future, there will be a demand for users wearing or holding a controller in their hand to be able to operate the controller for a long period of time in a comfortable position.

[0006] The present invention has been made in consideration of such problems, and aims to enable a user who wears or holds a controller in their hand to operate the controller for long periods of time in a comfortable position. [Means for solving the problem]

[0007] The information processing device of the present invention comprises a display means having a display screen visible to a user, a setting means for setting real space coordinates to be the home position of a controller worn or held in the user's hand based on position information and operation information transmitted from the controller, and a conversion means for converting the real space coordinates into virtual space coordinates within the range of the display screen. [Effects of the Invention]

[0008] According to the present invention, a user who wears or holds a controller in his or her hand can operate the controller for a long period of time in a comfortable position. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the appearance of a controller illustrated in FIG. [Figure 3]10 is a flowchart illustrating an example of a processing procedure in a control method for an information processing device according to an embodiment of the present invention. [Figure 4] 4 is a diagram for explaining an example of processing in a flowchart of a control method for the information processing device shown in FIG. 3. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] 1 is a diagram showing an example of a schematic configuration of an information processing system 10 according to an embodiment of the present invention. As shown in FIG. 1, the information processing system 10 includes an information processing device 100, which is a head-mounted display (HMD) in this embodiment, and a controller 200.

[0012] The information processing device 100 shown in FIG. 1 is configured as a head-mounted display (HMD), which is a head-mounted display device that can be worn on a user's head. The information processing device 100 has a display unit 150 equipped with a display screen that the user can view. The display screen of the display unit 150 of the information processing device 100 displays various images and virtual objects, which are CG (computer graphics) content, to the user. This allows the user to experience virtual reality with their own eyes. The information processing device 100 also has a function of detecting the user's hand through communication with the controller 200, acquiring information related to the position and posture of the hand, and then applying the movement of the hand to a virtual object. This allows the user to see their hand in a virtual space and use that hand to intuitively operate a virtual object.

[0013] As shown in FIG. 1, the information processing device 100 includes a control unit 110, a nonvolatile memory 120, a working memory 130, an operation unit 140, a display unit 150, a position detection unit 160, and a communication unit 170.

[0014] The control unit 110 controls the operation of each component (120 to 170) of the information processing device 100 in accordance with input signals and programs. The control unit 110 may be configured with, for example, one or more processors such as a CPU or a GPU. Note that instead of the control unit 110 controlling the entire information processing device 100, the entire information processing device 100 may be controlled by a plurality of hardware devices sharing the processing load.

[0015] The non-volatile memory 120 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 110, data on the virtual space and virtual objects displayed on the display screen of the display unit 150, and the like.

[0016] The work memory 130 is used as an image display memory for the display unit 150, a work area for the control unit 110, and the like.

[0017] The operation unit 140 is a component that is operated by the user when giving instructions to the information processing device 100. The operation unit 140 includes, for example, a power button and operation buttons of the information processing device 100.

[0018] The display unit 150 is a display means for displaying a virtual space and a GUI (Graphical User Interface) for interactive operation on a display screen. The display unit 150 has a display screen made of, for example, a liquid crystal panel or an organic EL panel. When a user wears the information processing device 100, which is a head-mounted display (HMD), the display screen of the display unit 150 is configured to be placed in front of each of the user's eyes.

[0019] The position detection unit 160 detects the position and attitude of the information processing device 100, which is a head-mounted display (HMD). The position detection unit 160 includes, for example, a global positioning system (GPS) and an inertial measurement unit (IMU). The position detection unit 160 outputs information on the attitude of the information processing device 100 as attitude information to the working memory 130 via the control unit 110.

[0020] The communication unit 170 performs wireless communication with the controller 200. The communication unit 170 performs wireless communication conforming to, for example, Bluetooth (registered trademark). Note that the communication unit 170 may also be other wireless communication means such as wireless LAN communication conforming to the IEEE802.11 standard.

[0021] The controller 200 is a device for performing various controls on the information processing device 100, which is a head-mounted display (HMD), and has a function of performing wireless communication with the information processing device 100. When a user operates the controller 200, the controller 200 transmits user operation information to the information processing device 100, and the information processing device 100 is controlled accordingly.

[0022] FIG. 2 is a diagram showing an example of the appearance of the controller 200 shown in FIG. 2 is formed in a finger ring shape so that it can be worn on a user's finger. If controller 200 can be worn on a user's finger, the user can move their hand freely while holding controller 200.

[0023] The shape of the controller 200 is not limited to the ring-like shape shown in FIG. 2. For example, the controller 200 may be a glove-like shape that can be worn on the hand. The controller 200 may be in any shape that can be held in the user's hand or worn on the user's hand for ease of use. The number of controllers 200 is not limited to one as shown in FIG. 1. In this embodiment, it is also possible to configure a plurality of controllers 200 so that, for example, a controller 200 is worn on each of the user's left and right hands and the information processing device 100 can be controlled by the controllers 200 worn on each of the left and right hands.

[0024] Here, we return to the explanation of Figure 1 again. As shown in FIG. 1, the controller 200 includes a control unit 210, a nonvolatile memory 220, a working memory 230, an operation unit 240, a position detection unit 260, and a communication unit 270.

[0025] The control unit 210 controls the operation of each component (220 to 240, 260, 270) of the controller 200 in accordance with input signals and programs. The control unit 210 may be configured with, for example, one or more processors such as a CPU or a GPU. Note that instead of the control unit 210 controlling the entire controller 200, the entire controller 200 may be controlled by a plurality of hardware devices sharing the processing load.

[0026] The nonvolatile memory 220 is an electrically erasable and recordable nonvolatile memory, and stores programs to be executed by the control unit 210 and the like.

[0027] The work memory 230 is used as a work area for the control unit 210, etc.

[0028] The operation unit 240 is a component that is operated by the user when giving instructions to the controller 200. The operation unit 240 includes, for example, a power button and operation buttons of the controller 200.

[0029] The position detection unit 260 detects the position and attitude of the controller 200. The position detection unit 260 includes, for example, a Global Positioning System (GPS) or an Inertial Measurement Unit (IMU). The position detection unit 260 outputs information on the attitude of the controller 200 as attitude information to the working memory 230 via the control unit 210.

[0030] The communication unit 270 performs wireless communication with the information processing device 100, which is a head-mounted display (HMD). The communication unit 270 performs wireless communication conforming to Bluetooth (registered trademark), for example. Note that the communication unit 270 may be other wireless communication means such as wireless LAN communication conforming to the IEEE802.11 standard.

[0031] Fig. 3 is a flowchart showing an example of a processing procedure in a control method for the information processing device 100 according to the embodiment of the present invention. Note that each process in the flowchart shown in Fig. 3 is realized by a control unit 110 of the information processing device 100, which is a head-mounted display (HMD), controlling the operation of each component (120 to 170) of the information processing device 100 in accordance with an input signal or a program.

[0032] 3, a wireless communication connection is established between the information processing device 100, which is a head-mounted display (HMD), and the controller 200, and they are in a state where they can communicate wirelessly with each other at any time. At this time, the information processing device 100, which is a head-mounted display (HMD), is assumed to be worn on the user's head, allowing the user to view the display screen of the display unit 150, which is a display means. The controller 200 is assumed to be worn or held by the user's hand. Furthermore, the controller 200 is assumed to be in a state where it is continuously transmitting, at a regular interval, to the information processing device 100, information on the user's button operation, obtained by the operation unit 240, and information on the position and orientation of the controller 200, obtained by the position detection unit 260.

[0033] 3, control unit 110 generates a virtual space image by combining one or more virtual objects acquired from nonvolatile memory 120, and displays the generated virtual space image on the display screen of display unit 150. Control unit 110 controls the position, orientation, and size of the virtual object in the virtual space image based on the position information and orientation information acquired by position detection unit 160. In this way, control unit 110 generates a virtual space image in which the virtual object appears as if it were placed in real space.

[0034] 3, control unit 110 determines whether or not a target object to be operated has been selected by the user through an operation from controller 200 via communication unit 170. Here, the target object to be operated is one of the virtual objects displayed on the display screen of display unit 150 in step S101 of FIG. 3. The target object to be operated is selected, for example, by the user selecting the target object through a hand gesture or button operation on controller 200, and then the user further operating a menu through a button operation or hand gesture.

[0035] If it is determined in step S102 in FIG. 3 that the user has not selected an object to be operated (S102 / NO), the process waits in step S102 until the user selects an object to be operated.

[0036] Also, if the result of the determination in step S102 in FIG. 3 is that an object to be operated has been selected by the user (S102 / YES), the process proceeds to step S103. When the process proceeds to step S103 in FIG. 3, the control unit 110 stores in the work memory 130 the virtual space coordinates of the operation target object determined to have been selected in step S102.

[0037] Next, in step S104 of Fig. 3, control unit 110 determines whether a home position has been set by the user by operating controller 200 via communication unit 170. Here, the home position refers to the origin coordinates in real space of the position and orientation of controller 200, and is linked to the vicinity of the virtual space coordinates of the object to be operated stored in step S103 in step S106 of Fig. 3, which will be described later. Furthermore, the home position is set, for example, by the user moving controller 200 to desired real space coordinates, and then further operating a menu on controller 200 using button operations or hand gestures.

[0038] If it is determined in step S104 in FIG. 3 that the home position has not been set by the user (S104 / NO), the process waits in step S104 until the home position is set by the user.

[0039] Also, if the result of the determination in step S104 in FIG. 3 is that the home position has been set by the user (S104 / YES), the process proceeds to step S105. 3, the control unit 110 stores the real space coordinates of the home position determined to have been set in step S104 in the working memory 130. The control unit 110 performing the processes of S104 and S105 in Fig. 3 constitutes a setting means that sets the real space coordinates as the home position of the controller 200 based on the position information and operation information transmitted from the controller 200 worn or held by the user's hand.

[0040] Next, in step S106 of FIG. 3, the control unit 110 converts the real space coordinates of the home position stored in the working memory 130 in S105 into virtual space coordinates near the virtual space coordinates of the object to be operated stored in the working memory 130 in S103.

[0041] Fig. 4 is a diagram for explaining an example of processing in a flowchart in the control method of the information processing device 100 shown in Fig. 3. Fig. 4 shows a display screen 400 provided on the display unit 150. This display screen 400 is visible to a user who wears the information processing device 100, which is a head-mounted display (HMD), on his or her head. On this display screen 400, an operation target object 401 determined to have been selected in step S102 of Fig. 3 is displayed at the virtual space coordinates stored in the work memory 130 in step S103 of Fig. 3.

[0042] Outside the range of the display screen 400 in real space, the actual user's right hand 420R and left hand 420L, the controller 200R worn or held by the right hand 420R, and the controller 200L worn or held by the left hand 420L are arranged. In FIG. 4 , the coordinates of the user's right hand 420R and controller 200R arranged in real space are referred to as real space coordinates 421R, and the coordinates of the user's left hand 420L and controller 200L arranged in real space are referred to as real space coordinates 421L. In this embodiment, the real space coordinates 421R are assumed to be real space coordinates that represent the home positions of the user's right hand 420R and controller 200R arranged in real space. Similarly, the real space coordinates 421L are assumed to be real space coordinates that represent the home positions of the user's left hand 420L and controller 200L arranged in real space. Although the real space coordinates 421R and 421L are outside the range of the display screen 400 in real space, the information processing device 100, which is a head mounted display (HMD), recognizes their positions based on position information from the controllers 200R and 200L.

[0043] 3, the control unit 110 converts the real space coordinates 421R into virtual space coordinates 402R, which are virtual space coordinates within the range of the display screen 400 and are near the virtual space coordinates of the operation target object 401. Similarly, in step S106 of FIG. 3, the control unit 110 converts the real space coordinates 421L into virtual space coordinates 402L, which are virtual space coordinates within the range of the display screen 400 and are near the virtual space coordinates of the operation target object 401. Here, in the example shown in FIG. 4, a virtual right hand (420R) is displayed in the virtual space coordinates 402R within the range of the display screen 400, but a virtual controller (200R) may be displayed in addition to or instead of this virtual right hand. Similarly, in the example shown in FIG. 4, a virtual left hand (420L) is displayed at virtual space coordinates 402L within the range of the display screen 400, but a virtual controller (200L) may be displayed in addition to or instead of this virtual left hand.

[0044] In the example shown in FIG. 4, as a comparative example of the present invention, a user's right hand 410R and left hand 410L (including controllers worn or held by these hands) in a real space that coincides with the virtual space are also illustrated. Also, as a comparative example of the present invention, real space coordinates 411R (which coincide with the virtual space coordinates) that are the home position of the user's right hand 410R and real space coordinates 411L (which coincide with the virtual space coordinates) that are the home position of the user's left hand 410L are also illustrated. In this comparative example, in order to operate the operation target object 401 using the virtual right hand displayed at virtual space coordinates 402R and the virtual left hand displayed at virtual space coordinates 402L, the user must lift both hands to the positions of the real space coordinates 411R and 411L. This can be very tiring and painful for the user to operate the operation target object 401 for a long period of time.

[0045] In contrast, in an embodiment of the present invention, in step S106 of FIG. 3 , real space coordinates 421R and real space coordinates 421L are converted into virtual space coordinates 402R and virtual space coordinates 402L, respectively, near the virtual space coordinates of operation target object 401. As a result, the user can operate operation target object 401 as if their real hand were located at virtual space coordinates 402R and 402L near operation target object 401 in the virtual space, while keeping their real hand at the real space coordinates 421R and 421L. That is, in an embodiment of the present invention, a user wearing or holding controller 200 on their hand can operate controller 200 for a long period of time in a comfortable position. In this embodiment, control unit 110 that performs the processing of S106 of FIG. 3 constitutes conversion means that converts real space coordinates representing the home positions of the user's hand and controller 200 into virtual space coordinates within the range of display screen 400.

[0046] Here, we return to the explanation of FIG. When the process of step S106 in FIG. 3 is completed, the process proceeds to step S107. 3, the control unit 110 determines whether or not the user has performed an operation to cancel the coordinate transformation performed in step S106 by operating the controller 200 via the communication unit 170. Here, the cancellation operation is performed by, for example, a menu operation using a hand gesture or a button operation on the controller 200.

[0047] As a result of the determination in step S107 of FIG. 3, if the user has not performed an operation to cancel the coordinate transformation in step S106 (S107 / NO), the process waits in step S107 until the user performs an operation to cancel the coordinate transformation in step S106.

[0048] Also, if the result of the determination in step S107 in FIG. 3 is that the user has performed an operation to cancel the coordinate transformation in step S106 (S107 / YES), the process proceeds to step S108. When the process proceeds to step S108 in FIG. 3, the control unit 110 cancels the coordinate transformation performed in step S106.

[0049] When the process of step S108 in FIG. 3 is completed, the process of the flowchart in FIG. 3 is completed.

[0050] The information processing device 100 according to the embodiment of the present invention described above has a display unit 150 equipped with a display screen 400 visually recognized by a user, and performs the following processing. Specifically, the control unit 110 of the information processing device 100 sets real space coordinates as the home position of the controller 200 based on position information and operation information transmitted from the controller 200 worn or held by the user's hand (S104 and S105 in FIG. 3). Then, the control unit 110 of the information processing device 100 converts the real space coordinates as the home position of the controller 200 into virtual space coordinates within the range of the display screen 400 (S106 in FIG. 3). With this configuration, even if the user assumes a comfortable position that allows for long periods of operation, the user's hand or controller 200 worn or held by the user's hand can be kept within the range of display screen 400 of display unit 150. This allows a user who wears or holds controller 200 in their hand to operate controller 200 for long periods of time in a comfortable position.

[0051] In addition, in Figure 4, an example has been described in which a virtual hand is displayed on display screen 400 and hand gesture operations are performed, but it is also possible to display a virtual controller 200 on display screen 400 and perform operations other than hand gestures.

[0052] In addition, in this embodiment, a configuration has been described in which real space coordinates representing the home positions of both hands of the user are converted into virtual space coordinates in the vicinity of one operation target object 401 in the virtual space. However, the present invention is not limited to this configuration. For example, a configuration in which real space coordinates representing the home positions of each of the user's hands are converted into virtual space coordinates in the vicinity of different operation target objects in the virtual space is also applicable to the present invention.

[0053] In addition, in this embodiment, a configuration has been described in which real space coordinates representing the home position of the user's hand are converted into virtual space coordinates in the vicinity of the operation target object 401, which is a virtual object. However, the present invention is not limited to this configuration. For example, a configuration in which the real space coordinates representing the home position of the user's hand are converted into virtual space coordinates in virtual home position coordinates that serve as the reference for the above-mentioned home position in a virtual space within the range of the display screen 400 is also applicable to the present invention.

[0054] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. This program and a computer-readable storage medium storing the program are included in the present invention.

[0055] It should be noted that the above-described embodiments of the present invention are merely illustrative examples of the implementation of the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0056] The disclosure of this embodiment includes the following configuration, method, and program. [Configuration 1] a display means having a display screen that is visually recognized by a user; a setting means for setting real space coordinates as a home position of the controller based on position information and operation information transmitted from the controller worn or held by the user's hand; a conversion means for converting the real space coordinates into virtual space coordinates within the range of the display screen; An information processing device comprising: [Configuration 2] the display means displays a virtual object on the display screen, The conversion means sets coordinates in the vicinity of the virtual object as the virtual space coordinates. 2. The information processing device according to configuration 1, [Configuration 3] The conversion means sets virtual home position coordinates that serve as a reference for the home position in the virtual space as the virtual space coordinates. 2. The information processing device according to configuration 1, [Configuration 4] a plurality of the controllers are worn or held in the user's hands; the setting means sets the real space coordinates for each of the plurality of controllers; The conversion means converts the real space coordinates into the virtual space coordinates for each of the controllers. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] The display means displays the virtual hand on the display screen based on the virtual space coordinates. 5. The information processing device according to any one of configurations 1 to 4. [Configuration 6] The display means displays the virtual controller on the display screen based on the virtual space coordinates. 6. The information processing device according to any one of configurations 1 to 5. [Method 1] A method for controlling an information processing device having a display means with a display screen that is visually recognized by a user, comprising: a setting step of setting real space coordinates as a home position of the controller based on position information and operation information transmitted from the controller worn or held by the user's hand; a conversion step of converting the real space coordinates into virtual space coordinates within the display screen; 1. A method for controlling an information processing device, comprising: [Program 1] A program for causing a computer to execute each step of the control method for an information processing device according to Method 1. [Explanation of symbols]

[0057] 10: Information processing system, 100: Information processing device, 110: Control unit, 120: Non-volatile memory, 130: Working memory, 140: Operation unit, 150: Display unit, 160: Position detection unit, 170: Communication unit, 200: Controller, 210: Control unit, 220: Non-volatile memory, 230: Working memory, 240: Operation unit, 260: Position detection unit, 270: Communication unit, 400: Display screen, 401: Object to be operated, 402: Virtual space coordinates, 410: User's hand, 411: Real space coordinates, 420: User's hand, 421: Real space coordinates

Claims

1. a display means having a display screen that is visually recognized by a user; a setting means for setting real space coordinates as a home position of the controller based on position information and operation information transmitted from the controller worn or held by the user's hand; a conversion means for converting the real space coordinates into virtual space coordinates within the range of the display screen; An information processing device comprising:

2. the display means displays a virtual object on the display screen, The conversion means sets coordinates in the vicinity of the virtual object as the virtual space coordinates.

2. The information processing apparatus according to claim 1, wherein:

3. The conversion means sets virtual home position coordinates that serve as a reference for the home position in the virtual space as the virtual space coordinates.

2. The information processing apparatus according to claim 1, wherein:

4. a plurality of the controllers are worn or held in the user's hands; the setting means sets the real space coordinates for each of the plurality of controllers; The conversion means converts the real space coordinates into the virtual space coordinates for each of the controllers.

2. The information processing apparatus according to claim 1, wherein:

5. The display means displays the virtual hand on the display screen based on the virtual space coordinates.

2. The information processing apparatus according to claim 1, wherein:

6. The display means displays the virtual controller on the display screen based on the virtual space coordinates.

2. The information processing apparatus according to claim 1, wherein:

7. A method for controlling an information processing device having a display means with a display screen that is visually recognized by a user, comprising: a setting step of setting real space coordinates as a home position of the controller based on position information and operation information transmitted from the controller worn or held by the user's hand; a conversion step of converting the real space coordinates into virtual space coordinates within the display screen; 1. A method for controlling an information processing device, comprising:

8. A program for causing a computer to execute each step of the method for controlling an information processing device according to claim 7.

Images