Information processing method, information processing apparatus, and program

Abstract

To provide an operating method according to the state of a user by changing IMU data according to the state of the user.SOLUTION: An information processing method for controlling a user's operation input includes: a data acquisition step of acquiring IMU data from an inertial sensor attached to the user; a display step of displaying the user's operating state according to the acquired IMU data; an information acquisition step of acquiring the user's biological information; a determination step of determining the state of the user according to the acquired biological information; and a processing step of changing the IMU data used in the display step according to the determined state of the user.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing method, an information processing device, and a program, and more particularly to an information processing method for operation input utilizing data from an inertial sensor. [Background technology]

[0002] In recent years, the use of inertial sensors (Inertial Measurement Units, hereafter referred to as IMUs) in control devices has been increasing.

[0003] For example, with a Head Mounted Display (HMD), traditional operating devices such as a keyboard and mouse cannot be used, so controllers are attached to the hands and fingers. When these controllers are moved in real space, the IMU built into the controller detects the controller's movement and reflects the controller's movement in the image displayed by the HMD. Specifically, it displays a cursor (pointer) and a ray to select an object to be operated on the image displayed by the HMD.

[0004] Meanwhile, there is known a technology that incorporates various sensors in a device worn by a user to measure heart rate, blood oxygen saturation, sweating, body temperature, etc., acquires the user's biological information, and estimates the user's activity and mental state. For example, Patent Document 1 discloses a device that determines the degree of fatigue from biological information, and Patent Document 2 discloses a device that determines the cause of stress. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6676499 [Patent Document 2] JP 2019-209128 A [Patent Document 3] Patent No. 5346115 [Patent Document 4] JP 2017-120550 A [Non-patent literature]

[0006] [Non-Patent Document 1] Z.Zhang. “A flexible new technique for camera calibration”. Technical Report MSR-TR-98-71 1998. [Non-Patent Document 2] H. Kato and M. Billinghurst. “Marker tracking and hmd calibration for a video-based augmented reality conferencing system”. International Workshop on Augmented Reality,1999. [Non-Patent Document 3] Raul Mur-Artal and Juan D. Tardos. “ORB-SLAM2:An Open-Source SLAM System for Monocular,Stereo,and RGB-D Cameras”. IEEE Transactions on Robotics(Volume:33,Issue:5,Oct.2017). Summary of the Invention [Problem to be solved by the invention]

[0007] When using an IMU-based operating device, there is a problem that operability may decrease depending on the user's state (activity state, mental state). For example, if a user is in an excessively tense or irritated state, their body movements will be different from normal, and the way they move the operating device will also change.

[0008] To address such problems, Patent Document 3 discloses a method for changing a guide displayed as AR (Augmented Reality) according to the user's biometric information. This prior art allows display suitable for the user's state, improving operability. Patent Document 4 discloses a method for dynamically switching input methods according to the user's activity state. This prior art allows use of an input method suitable for the user's state, improving operability.

[0009] However, even if these prior art technologies are provided, there are problems as follows: If the guide displayed in AR is changed according to the user's state or the input method is changed, the user cannot use the same operation method. [Means for solving the problem]

[0010] Therefore, an information processing method as one aspect of the present invention is characterized by having a data acquisition step of acquiring IMU data from an inertial sensor worn by a user, a display step of displaying the user's operation status based on the acquired IMU data, an information acquisition step of acquiring the user's biometric information, a determination step of determining the user's status based on the acquired biometric information, and a processing step of changing the IMU data used in the display step based on the determined user's status.

[0011] Other aspects of the present invention will become apparent from the embodiments described below. Effect of the Invention

[0012] According to the present invention, by changing IMU data according to the state of the user, it is possible to provide an operation method according to the state of the user. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating the configuration of an HMD device and a controller to which an information processing method of the present invention can be applied. [Diagram 2] 1 is a diagram for explaining the external appearance of an HMD device to which an information processing method of the present invention can be applied. [Diagram 3] 2 is an example of an image displayed on an HMD device (embodiment 1) to which the information processing method of the present invention can be applied. [Figure 4] 4 is a flowchart for explaining the operation of the HMD device (embodiment 1) to which the information processing method of the present invention can be applied. [Diagram 5] 13 is an example of an image displayed on an HMD device (embodiment 2) to which the information processing method of the present invention can be applied. [Figure 6] 11 is a flowchart for explaining the operation of an HMD device (embodiment 2) to which the information processing method of the present invention can be applied. [Figure 7] 13 is an example of an image displayed on an HMD device (embodiment 3) to which the information processing method of the present invention can be applied. [Figure 8] 11 is a flowchart for explaining the operation of an HMD device (third embodiment) to which the information processing method of the present invention can be applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0015] Example 1 In this embodiment, a case will be described in which the information processing method of the present invention is applied to a system including a head mounted display device (hereinafter, referred to as an HMD device) and its controller.

[0016] In this embodiment, the HMD device will be described as a device for experiencing MR (Mixed-Reality), but it may also be an HMD device for AR (Augmented-Reality) or VR (Virtual-Reality).

[0017] <Device Description> FIG. 1 is a block diagram for explaining a system including an HMD device and a controller to which the present invention can be applied.

[0018] In Fig. 1, A100 is an HMD device. A101 to A113 are hardware resources that constitute the HMD device A100. A plurality of devices are connected to a system bus A113. Specifically, they are a CPU A101, a RAM A102, a ROM A103, a GPU A104, an operation unit A105, a short-range communication unit A016, an imaging unit A107, a display unit A108, a sensor unit A109, a battery A110, a large-capacity storage unit A111, and a communication unit A112.

[0019] A101 is a CPU and is responsible for controlling the entire HMD device A100. The CPU A101 controls each unit described later, and performs operations according to inputs from an operation unit A105, image data captured by an imaging unit A107, data received from a short-range communication unit A106, and the like.

[0020] The RAM A102 is a rewritable memory, and is used as a work area by the program that controls the HMD device A100. The RAM A102 is, for example, a volatile memory (DRAM) that uses semiconductor elements.

[0021] A103 is a non-volatile memory, and holds a program that controls the HMD device A100. When the power is turned on to the HMD device A100, the CPU A101 reads the program from the ROM A103 and starts controlling the HMD device A100. The ROM A103 is, for example, a flash memory.

[0022] A104 is a GPU, a unit used for parallel data processing. It uses an LSI called a Graphics Processing Unit, which is designed to perform various calculations in parallel at high speed for rendering 3D model data, but equivalent functions can also be achieved with a reconfigurable logic circuit called an FPGA.

[0023] A105 is an operation unit that is used to transmit user instructions to the HMD device 100. The operation unit 105 is composed of buttons, dials, and the like.

[0024] A106 is a short-distance communication unit, which is a unit for realizing communication with a short-distance communication unit B108 of the controller B100 (described later). The short-distance communication unit A106 is composed of an antenna and a communication controller for realizing wireless communication according to, for example, Bluetooth.

[0025] A107 is an imaging unit consisting of a stereo camera, which captures color images of a scene using the two cameras mounted on the left and right sides and outputs video signals to the internal bus A113. This video signal undergoes various image processing by the CPU A101 and is stored in the RAM A102 as left and right image data. The imaging unit A107 is composed of, for example, an optical system that controls an optical lens unit and aperture, zoom, focus, etc., and an imaging element for converting light (image) introduced through the optical lens unit into an electrical video signal. As the imaging element, a CMOS imaging element (CMOS image sensor) using a CMOS or a CCD imaging element (CCD image sensor) using a CCD is generally used.

[0026] A108 is a display unit consisting of two display devices, which displays an image synthesized by the CPU A101 between the 3D model data rendered by the GPU A104 and the left and right image data captured by the imaging unit A107 and stored in the RAM A102. By displaying different images that take parallax into consideration on the left and right display devices, it is possible to provide a stereoscopic view of the real space and the 3D model data.

[0027] A109 is a sensor unit, and is composed of a sensor unit other than an image sensor. For example, it is composed of an acceleration sensor for detecting a change in the position of the HMD device A100, a gyro sensor for detecting a change in posture, a geomagnetic sensor for detecting a direction, etc. In addition, it may include a ToF (Time of Flight) sensor for measuring distance and a LiDAR scanner for acquiring the geometric shape of space.

[0028] A110 is a battery, and is a unit for supplying power for operating the HMD device A100. The battery A110 is a unit made up of a rechargeable secondary battery, and can be charged by an external battery charger (not shown).

[0029] A111 is a rewritable and non-volatile storage unit, and holds a program for controlling the HMD device A100, various data used by the program, data acquired from a communication unit A112 described later, etc. The large-capacity storage unit A111 is composed of a semiconductor storage device such as an eMMC (embedded multi media card) or an SSD (solid state disk), etc. The large-capacity storage unit A111 may also be composed of a storage medium such as a memory card and an interface unit for mounting the storage medium.

[0030] A112 is a communication unit, which is a unit for realizing communication with an external device (not shown) (for example, a server on the Internet or a personal computer on a local network). The communication unit 111 is composed of an antenna and a communication controller for realizing wireless communication according to, for example, IEEE802.11a / b / g / n / ac / ax.

[0031] A113 is a system bus, and each unit connected to the system bus A113 is capable of exchanging data with each other via the system bus A113.

[0032] In FIG. 1, B100 is a controller for operating the HMD device A100.

[0033] B101 to B109 are hardware resources that constitute the controller B100. A CPU B101, a RAM B102, a ROM B103, an IMU B104, an operation unit B105, a sensor unit B106, a battery B107, and a short-range communication unit B108 are connected to the system bus B109.

[0034] B101 is a CPU, and is responsible for controlling the entire controller B100. The CPU B101 controls each unit described below, and performs operations according to inputs from an operation unit B105 and data received from a short-range communication unit B108.

[0035] B102 is a rewritable memory, and is used as a work area for the program that controls the controller B100. The RAM B102 is, for example, a volatile memory (DRAM) that uses semiconductor elements.

[0036] B103 is a non-volatile memory, and holds a program that controls the controller B100. When power is applied to the controller B100, the CPU B101 reads the program from the ROM B103 and starts controlling the controller B100. The ROM B103 is, for example, a flash memory.

[0037] Although the CPU B101, RAM B102, and ROM B103 have been described here as separate hardware resources, these functions may be integrated into a single LSI.

[0038] B104 is an inertial sensor unit (IMU: Inertial Measurement Unit) that is used to detect changes in the physical position and attitude of the controller B100. For example, it is composed of an acceleration sensor for detecting speed changes on three axes (up, down, left, right, front, back) and an angular velocity center (gyro sensor) for detecting rotation speed.

[0039] B105 is an operation unit that is used to transmit user instructions to the controller B100. The operation unit B105 is composed of buttons, dials, touch sensors, and the like.

[0040] B106 is a sensor unit that is used to acquire biometric information of the user who is using the controller B100. The sensor unit B106 includes a heart rate sensor that measures the heart rate, a temperature sensor that measures the body temperature, an optical sensor that measures the oxygen saturation level in the blood, a humidity sensor that measures the degree of sweating, and the like.

[0041] B107 is a battery, which is a unit for supplying power for operating the controller B100. The battery B107 is a unit made up of a rechargeable secondary battery, and can be charged by an external battery charger (not shown).

[0042] B108 is a short-distance communication unit, which is a unit for realizing communication with the short-distance communication unit A106 of the HMD device A100. The short-distance communication unit B108 is composed of an antenna and a communication controller for realizing wireless communication according to, for example, Bluetooth.

[0043] B109 is a system bus, and each unit connected to the system bus B109 can exchange data with each other via the system bus B109.

[0044] FIG. 2A is a diagram showing the HMD device A100 of this embodiment as seen from the outside.

[0045] 2A, 201 and 202 are cameras (lenses) constituting the imaging unit A107. In this embodiment, the imaging unit A107 is composed of a background stereo camera 201 for obtaining image data for displaying an image of real space on the display unit A108, and a stereo camera 202 for obtaining image data for estimating the self-position and orientation. However, the imaging unit A107 may be composed of one stereo camera that serves both as the background stereo camera and the position estimation stereo camera.

[0046] 203 is an area for arranging the sensor unit A109. When the sensor unit A109 has a ToF sensor or LiDAR, a sensor for emitting a laser and receiving reflected light is arranged in 203.

[0047] FIG. 2B is a diagram showing the HMD device A100 of this embodiment as seen from the inside.

[0048] In Fig. 2B, 204 is a display device constituting the display unit A108, which is composed of two display devices to display different images to the left and right eyes of the wearer. These display devices are, for example, small organic EL displays or liquid crystal displays.

[0049] FIG. 2C is a diagram for explaining the external appearance of the controller B100 of this embodiment.

[0050] In this embodiment, the controller B100 is stick-shaped and can be operated by a user by holding it in his / her hand. The controller B100 is also provided with buttons 205 constituting the operation unit B105, and the user can also use these buttons to perform operations.

[0051] <Operation description> The operation of the HMD device A100 and the controller B100 to which the information processing method of the present invention is applied will be described. First, an example of a screen displayed on the display unit A108 of the HMD device A100 will be introduced using Fig. 3, and the operation for realizing that screen will be described using the flowchart in Fig. 4.

[0052] Fig. 3 is a diagram for explaining how the HMD device A100 is operated using the controller B100. Fig. 3(A) to (D) show images displayed on the display unit A108 of the HMD device A100. The display unit A108 consists of two display devices as shown in Fig. 2(B), but for simplicity, it will be explained here using one diagram.

[0053] 3A is an example of an image displayed on the display unit A108. Here, computer graphics (CG) of an automobile shown in 301 is being confirmed using the HMD device A100. Also, a setting GUI for changing the CG is displayed in 302.

[0054] 3B shows a state in which the user starts to operate the setting GUI 302 using the controller B100. When the user wearing the HMD device A100 lifts the controller B100, the IMU B104 of the controller B100 detects the movement. Then, the controller and the hand are displayed in CG on the screen as shown by 303.

[0055] 3(C) shows how the user displays a ray using the operation unit B105 of the controller B100. When the user presses a button that constitutes the operation unit B105, a CG image of the ray is displayed from above the controller displayed in CG, as shown in 304. At this time, the direction of the ray 304 is aligned with the attitude of the controller B100 measured by the IMU B104.

[0056] FIG. 3(D) shows how the user physically moves the controller B100 and operates the GUI302. When the user physically moves the controller B100, the IMU B104 measures the movement, and the CG controller also moves as shown in 305 (here, it is moved to the left and tilted to the left). At the same time, the CG of the light ray emitted from the controller also moves as shown in 306. When the light ray 306 overlaps with the GUI302, the item in that part is highlighted as shown in 307. When the user presses a button constituting the operation unit B105, this item is selected and the function is executed. For example, the color of the CG of the car 301 can be changed.

[0057] By using the controller B100 in this manner, the HMD device A100 can be operated in accordance with the physical movement of the controller B100.

[0058] FIG. 4 is a flowchart for explaining the operation of the HMD device A100.

[0059] 4, the processes of steps S403 to S417 are repeatedly executed in accordance with the imaging timing of the imaging unit A 107. For example, if the imaging unit A 107 can capture 60 frames per second, the processes of steps S403 to S417 are also repeated 60 times per second.

[0060] In step S401, the CPU A101 performs an initialization process. In the initialization process, for example, the CPU A101 reads the internal parameters of the imaging unit A107 and calculates the initial position / orientation of the HMD device A100. The internal parameters of the imaging unit A107 (focal length, image center, lens distortion parameters, and angle of view) are calibrated in advance by the Zhang method (Non-Patent Document 1). The relative position / orientation (external parameters) between the two cameras constituting the imaging unit A107, which is a stereo camera, is obtained, for example, from images obtained by simultaneously photographing a pattern whose three-dimensional shape is known with the left and right cameras. That is, the relative position / orientation between the two cameras is obtained by determining the position / orientation of each camera based on the known pattern and converting the position / orientation of the right camera into the position / orientation based on the left camera. The initial position / orientation of the HMD device A100 (initial value of the position / orientation relative to the world coordinate system) can be calculated, for example, by the method of Kato et al. (Non-Patent Document 2) using an artificial marker whose size is known.

[0061] In step S402, the CPU A101 communicates with a server (not shown) via the communication unit A112 to acquire CG data to be used in the HMD device A100. For example, data for rendering CG301 in Fig. 3A is acquired. Note that the CG data may be stored in the mass storage unit A112 in advance, and the data from the mass storage unit A112 may be acquired in this step.

[0062] In step S403, the CPU A101 estimates the position and orientation of the HMD device A100 in the real space. This process is called SLAM (Simultaneous Localization and Mapping), and various algorithms are known for this process. For example, SLAM (Visual SLAM) using stereo images can be realized by the method of Mur-Artal et al. (Non-Patent Document 3). Note that other generally known SLAM algorithms may be used as the SLAM algorithm. For example, monocular SLAM using one camera, or Visual-Inertial SLAM combining an inertial sensor and a stereo camera may be used. In addition, RGBD SLAM combining a depth sensor such as a ToF sensor and a stereo camera, Lidar SLAM using Lidar, SLAM using machine learning, and the like may also be used.

[0063] In step S404, the CPU A101 renders the CG model data acquired in step S402 based on the self-position and orientation estimated in step S403. In the example of Fig. 3, 301 corresponds to this rendering result. This rendering process is realized in cooperation with the GPU A104. The CPU A101 stores the image data obtained by rendering in the RAM A102.

[0064] In step S405, the CPU A101 acquires values ​​(acceleration and angular velocity) of the IMU B104 of the controller B100 using the short-range communication unit A106 (data acquisition step). This process is realized by the CPU B101 of the controller B100 acquiring data of the IMU B104 and responding to the IMU data acquisition request received by the short-range communication unit B108.

[0065] In step S406, the CPU A101 acquires values ​​(heart rate, body temperature, etc.) of the sensor unit B106 of the controller B100 using the short-range communication unit A106 (information acquisition step). This process is realized by the CPU B101 of the controller B100 acquiring data from the sensor unit B106 and responding to the biometric information acquisition request received by the short-range communication unit B108.

[0066] In step S407, the CPU A101 judges the state of the user operating the controller B100 based on the biological information acquired in step S406 (judgment step). For example, the CPU A101 judges the user's level of excitement, tension, stress, etc. For the judgment, a generally known algorithm in addition to those disclosed in Patent Documents 1 and 2 may be used.

[0067] In step S408, the CPU A101 judges whether the user state judged in step S407 is a predetermined state. For example, if the degree of excitement or tension is clearly high, the result in this step is Yes. If the result in this step is Yes, the process proceeds to step S409, and if the result is No, the process proceeds to step S410.

[0068] In step S409, the CPU A101 processes the IMU data (acceleration and angular velocity) acquired in step S405 according to the user state determined in step S407 (processing step). For example, if the degree of excitement determined in step S407 is obviously higher than normal, the CPU A101 operates to multiply the IMU data by a predetermined coefficient (such as 0.8) to make the acceleration and angular velocity smaller than the actually acquired values. In this way, the movement of the controller B100 can be treated as a smaller movement than the actual movement. Alternatively, if the degree of tension determined in step S407 is obviously higher than normal, the CPU A101 operates to lower the cutoff frequency of a low-pass filter for removing noise from the IMU data and remove a wider range of noise than normal. In this way, even if a minute vibration is applied to the controller B100 due to hand tremors or the like, the vibration can be ignored.

[0069] In step S410, the CPU A101 calculates the position and orientation of the controller B100 using the IMU data processed in step S409 (Yes in step S408) or the IMU data acquired in step S405 (No in step S408). Specifically, the position is calculated from the forward / backward, up / down, left / right movement distances obtained by integrating the acceleration twice, and the orientation is calculated from the amount of rotation obtained by integrating the angular velocity once.

[0070] In step S411, the CPU A101 acquires the state of the operation unit B105 of the controller B100 using the near-field communication unit A106. This process is realized by the CPU B101 of the controller B100 acquiring the state of the operation unit B105 and responding to the operation unit state acquisition request received by the near-field communication unit B108.

[0071] In step S412, the CPU A101 renders CG according to the position and orientation of the controller B100 updated in step S410 and the state of the operation unit B105 acquired in step S411 (display step). This CG represents the user's operating status, such as the controller, hand, and light beam. In the example of FIG. 3, 303 to 306 correspond to the rendering results. This rendering process is realized in cooperation with the GPU A104. The CPU A101 stores the image data obtained by rendering in the RAM A102.

[0072] In step S413, the CPU A101 renders, in CG, a GUI for the user to operate the HMD device A100 (UI display step). In the example of Fig. 3, 302 and 307 correspond to the rendering result. This rendering process is realized in cooperation with the GPU A104. The CPU A101 stores the image data obtained by rendering in the RAM A102.

[0073] In step S414, the CPU A101 obtains the results of rendering in steps S404, S412, and S413 from the RAM A102 and synthesizes them. The synthesized result is then displayed as a stereo image on the display unit A108. Note that examples of the images displayed on the display unit A108 here are the previously described FIGS. 3(A) to 3(D).

[0074] In step S415, the CPU A101 determines whether or not a GUI has been selected based on the position and orientation of the controller B100 updated in step S410 and the state of the operation unit B105 acquired in step S411. For example, when an item where the light ray 306 and the GUI 302 overlap is highlighted as in 307 as in FIG. 3D, and the user presses a button constituting the operation unit B105, it is determined that the item has been selected. If a GUI has been selected, the process proceeds to step S416, and if not, the process proceeds to step S417.

[0075] In step S416, the CPU A101 performs processing according to the GUI selected in step S415. For example, in the state shown in Fig. 3D, the item 307 performs processing to change the color of the automobile CG 301.

[0076] In step S417, the CPU A101 judges whether or not to terminate the system. If an instruction to terminate the system is given by the user via the operation unit A105, the system is terminated, otherwise the process returns to step S403 and continues processing the next frame. Note that an instruction to terminate the system may be given from an external device (not shown) via the communication unit A112.

[0077] By the above process, the IMU data can be changed according to the user's state obtained from the biometric information in this embodiment, and an operation method according to the user's state can be provided. For example, if the user is excited, the IMU data can be multiplied by a predetermined coefficient to make the movement smaller than the actual movement, or if the user is tense, fine vibrations can be removed, thereby providing an operation method according to the user's state.

[0078] Example 2 In this embodiment, an example in which the information processing method of the present invention is applied to an HMD device and its controller, similarly to the first embodiment, will be described.

[0079] <Device Description> Since this is similar to the first embodiment, the explanation will be omitted.

[0080] <Operation description> The operation of the HMD device A100 and the controller B100 to which the information processing method of the present invention is applied will be described. Here, an example of a screen displayed on the display unit A108 of the HMD device A100 will be introduced using Fig. 5, and the operation for realizing that screen will be described using the flowchart in Fig. 6.

[0081] Fig. 5 is a diagram for explaining how the HMD device A100 is operated using the controller B100. Fig. 5(A) to (D) show images displayed on the display unit A108 of the HMD device A100. The display unit A108 consists of two display devices as shown in Fig. 2(B), but for simplicity, it will be explained here using one diagram.

[0082] FIG. 5(A) is similar to FIG. 3(A), and therefore the description will be omitted.

[0083] FIG. 5B is similar to FIG. 3B, and therefore the description will be omitted.

[0084] 5C shows how the user displays a pointer using the operation unit B105 of the controller B100. When the user presses a button that constitutes the operation unit B105, a pointer (x) is displayed on the screen. At this time, the position of the pointer 504 is displayed in a direction that follows the attitude of the controller B100 measured by the IMU B104.

[0085] FIG. 5(D) shows how the user physically moves the controller B100 and operates the GUI 502. When the user physically moves the controller B100, the IMU B104 measures the movement, and the CG controller also moves as shown in 505 (here, it moves to the left and is tilted to the left). At the same time, a pointer 506, which is displayed in a direction that is aligned with the orientation of the controller, also moves. When the pointer 506 overlaps with the GUI 502, the item in that area is highlighted as shown in 504. When the user presses a button constituting the operation unit B105, this item is selected and the function is executed. For example, the color of the CG of the car 501 can be changed.

[0086] By using the controller B100 in this manner, the HMD device A100 can be operated in accordance with the physical movement of the controller B100.

[0087] FIG. 6 is a flowchart for explaining the operation of the HMD device A100.

[0088] 6, the processes of steps S604 to S621 are repeatedly executed in accordance with the imaging timing of the imaging unit A 107. For example, if the imaging unit A 107 can capture 60 frames per second, the processes of steps S604 to S621 are also repeated 60 times per second.

[0089] Step S601 is similar to step S401.

[0090] In step S602, the CPU A101 initializes T1, which is a variable used in the processing described later.

[0091] Steps S603 to S608 are similar to steps S402 to S407.

[0092] In step S609, the CPU A101 judges whether the user state judged in step S608 is a predetermined state. For example, if the degree of excitement or tension is clearly high, the result in this step is Yes. If the result in this step is Yes, the process proceeds to step S610, and if the result is No, the process proceeds to step S613.

[0093] In step S610, the CPU A101 increments the variable T1.

[0094] In step S611, the CPU A101 determines whether the value of the variable T1 is higher than a predetermined threshold value. This makes it possible to determine whether the user state determined in step S609 has continued for a predetermined time. For example, if the imaging unit A100 can capture 60 frames per second, then if the threshold value is set to 60, it will determine whether the predetermined state determined in step S609 has continued for one second. If the value of the variable T1 is higher than the predetermined threshold value, the process proceeds to step S612; otherwise, the process proceeds to step S614.

[0095] Step S612 is similar to step S409.

[0096] On the other hand, in step S613, the CPU A101 initializes the variable T1, which results in the variable T1 being initialized when the user state determined in step S609 has changed.

[0097] Steps S614 to S615 are similar to steps S410 to S411.

[0098] In step S616, the CPU A101 renders the controller, hand, pointer, etc. in CG according to the position and orientation of the controller B100 updated in step S614 and the state of the operation unit B105 acquired in step S615. In the example of FIG. 5, 503 to 506 correspond to this rendering result. This rendering process is realized in cooperation with the GPU A104. The CPU A101 stores the image data obtained by rendering in the RAM A102.

[0099] Steps S617 and S618 are similar to steps S413 and S414.

[0100] In step S619, the CPU A101 determines whether or not a GUI has been selected based on the position and orientation of the controller B100 updated in step S614 and the state of the operation unit B105 acquired in step S615. For example, when an item where the pointer 506 and GUI 502 overlap is highlighted as in 507 as in Fig. 5(D) and the user presses a button constituting the operation unit B105, it is determined that the item has been selected. If a GUI has been selected, the process proceeds to step S620, and if not, the process proceeds to step S621.

[0101] In step S620, the CPU A101 performs processing according to the GUI selected in step S619. For example, in the state of Fig. 5D, the item indicated by 307 performs processing to change the color of the automobile CG 501.

[0102] Step S621 is similar to step S417.

[0103] Through the above processing, in this embodiment, if the user's condition obtained from biometric information is maintained for a predetermined period of time, the IMU data can be changed according to the user's condition, and an operation method according to the user's condition can be provided.

[0104] Example 3 In this embodiment, an example in which the information processing method of the present invention is applied to an HMD device and its controller, similarly to the first embodiment, will be described.

[0105] <Device Description> Since this is similar to the first embodiment, the explanation will be omitted.

[0106] <Operation description> The operation of the HMD device A100 and the controller B100 to which the information processing method of the present invention is applied will be described. Here, an example of a screen displayed on the display unit A108 of the HMD device A100 will be introduced using Fig. 7, and the operation for realizing that screen will be described using the flowchart in Fig. 8.

[0107] Fig. 7 is a diagram for explaining how the HMD device A100 is operated using the controller B100. Fig. 7(A)-(D) show images displayed on the display unit A108 of the HMD device A100. The display unit A108 is composed of two display devices as shown in Fig. 2(B), but for simplicity, it will be explained here using one diagram.

[0108] 7(A) to (C) are similar to FIGS. 3(A) to (C), and therefore the description will be omitted.

[0109] 7(D) shows what happens when the user physically moves the controller B100 to operate the GUI 702. When the user physically moves the controller B100, the IMU B104 measures that movement, and the CG controller also moves as shown in 705 (here, it is moved to the left and tilted to the left). At the same time, the CG of the light beam emitted from the controller also moves as shown in 706. If the user moves too much, the light beam will exceed the GUI 702 as shown in 706, making it impossible to operate.

[0110] 7(E) shows the state when the user physically moves the controller B100 again to operate the GUI 702. When the controller is moved to the right as shown in 707 and tilted leftward, the CG of the light ray also moves as shown in 708. When the light ray 708 overlaps with the GUI 702, the item in that portion is highlighted as shown in 709. When the user presses a button constituting the operation unit B105 at this point, this item is selected and the function is executed. For example, the color of the CG of the car in 701 can be changed.

[0111] FIG. 8 is a flowchart for explaining the operation of the HMD device A100.

[0112] 8, the processes of steps S804 to S822 are repeatedly executed in accordance with the imaging timing of the imaging unit A 107. For example, if the imaging unit A 107 can capture 60 frames per second, the processes of steps S804 to S822 are also repeated 60 times per second.

[0113] Step S801 is similar to step S401.

[0114] In step S802, the CPU A101 initializes T2, which is a variable used in the process described below.

[0115] Steps S803 to S808 are similar to steps S402 to S407.

[0116] In step S809, the CPU A101 judges whether the user state judged in step S809 is a predetermined state. For example, if the degree of excitement or tension is clearly high, the result in this step is Yes. If the result in this step is Yes, the process proceeds to step S810, and if the result is No, the process proceeds to step S812.

[0117] In step S810, the CPU A101 judges whether the value of the variable T2 is higher than a predetermined threshold value. This makes it possible to judge whether the state judged in step S819 described below (a state in which the CG of the light beam points near the GUI) has continued for a predetermined time. For example, if the imaging unit A100 can capture 60 frames per second, then if the threshold value is set to 60, it will be judged whether the predetermined state judged in step S819 has continued for one second. If the value of the variable T2 is higher than the predetermined threshold value, the process proceeds to step S811; otherwise, the process proceeds to step S812.

[0118] Steps S811 to S816 are similar to steps S409 to S414.

[0119] In step S817, the CPU A101 determines whether or not the GUI has been selected, as in step S415. For example, when an item in the overlapping portion of the light ray 708 and the GUI 702 is highlighted as in 709 as in Fig. 7E, and the user presses a button constituting the operation unit B105, it is determined that the item has been selected. If the GUI has been selected, the process proceeds to step S818, and if not, the process proceeds to step S819.

[0120] Step S818 is similar to step S416.

[0121] In step S819, the CPU A101 judges whether or not the CG of the ray emitted from the controller B100 based on the position and orientation updated in step S812 points near the GUI. For example, if the ray points near the GUI 702, like the ray 704 in Fig. 7C or the ray 706 in Fig. 7D, the result of this judgment is Yes. Note that the judgment as to whether or not the ray is near the GUI can be made by judging whether or not the distance (Euclidean distance in pixels) between the central pixel of the ray and the closest GUI is equal to or less than a predetermined value.

[0122] In step S820, the CPU A101 increments the variable T2 (timer step).

[0123] On the other hand, in step S821, the CPU A101 initializes the variable T2. As a result, the variable T2 is initialized when the GUI is selected (Yes in step S817) and when the light beam leaves the vicinity of the GUI (No in step S819), which will be described later.

[0124] Step S822 is similar to step S417.

[0125] By the above processing, in this embodiment, when the user is trying to operate the displayed user interface for a predetermined time or more, the IMU data is changed according to the user's state, and an operation method according to the user's state can be provided. For example, when the user is excited and moves the controller a lot, making it impossible to select the displayed user interface as intended, the IMU data can be processed to reduce the movement of the light beam, making it easier to select.

[0126] (Other Examples) In the first to third embodiments, all the processes are performed by the HMD device A100, but some of the processes may be performed by an external device. For example, the CPU A101 determines the user state based on the acquired biometric information (steps S407, S608, and S808), but the CPU B101 of the controller B100 may determine the user state. In this case, the CPU A101 obtains the user state from the controller via the short-range wireless communication unit A106, and performs the subsequent processes (steps S408, S609, and S809).

[0127] In the first to third embodiments, the claimed information processing method is applied to an HMD device, but the present invention is not limited to this and may be applied to information processing devices of different forms, such as tablets and smartphones.

[0128] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0129] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0130] The disclosure of the present embodiment includes the following methods, configurations, and programs.

[0131] (Method 1) a data acquisition step of acquiring IMU data from an inertial sensor attached to a user; a display step of displaying a user's operation status according to the acquired IMU data; An information acquisition step of acquiring biometric information of a user; A determination step of determining a state of the user based on the acquired biometric information; An information processing method comprising: a processing step of changing the IMU data used in the display step depending on the determined state of the user.

[0132] (Method 2) The information processing method according to method 1, wherein the display step displays CG rays according to the IMU data.

[0133] (Method 3) The information processing method described in Method 1, characterized in that the display step displays a pointer according to IMU data.

[0134] (Method 4) The information processing method according to any one of Methods 1 to 3, wherein the processing step changes a cutoff frequency of a low-pass filter.

[0135] (Method 5) The information processing method described in any one of Methods 1 to 3, characterized in that in the processing step, the IMU data is multiplied by a predetermined coefficient.

[0136] (Method 6) The information processing method according to any one of Methods 1 to 5, wherein the judgment step judges the stress state of the user.

[0137] (Method 7) The information processing method according to any one of Methods 1 to 5, wherein the determining step determines a state of tension of the user.

[0138] (Method 8) The information processing method according to any one of Methods 1 to 7, wherein in the determining step, it is determined that the user state is that state if the user state is maintained for a predetermined period of time.

[0139] (Method 9) A UI display step of displaying a user interface for receiving a user operation, and a timer step of measuring time, An information processing method according to any one of methods 1 to 8, characterized in that the processing step is applied when an operation is attempted in the vicinity of the user interface for a predetermined period of time or longer in the display step.

[0140] (composition) A data acquisition means for acquiring IMU data from an inertial sensor attached to the user; A display means for displaying a user's operation status according to the acquired IMU data; An information acquisition means for acquiring biometric information of a user; A determination means for determining a state of a user based on the acquired biometric information; and processing means for changing the IMU data used by the display means according to the determined state of the user.

[0141] (program) A program for causing a computer to execute each step of the information processing method according to any one of Methods 1 to 9. [Explanation of symbols]

[0142] A100 HMD device to which the information processing method of the present invention can be applied B100: A controller for operating the HMD device A100

Claims

1. A data acquisition step of acquiring IMU data from an inertial sensor attached to a user; A display step of displaying an operation status of a user according to the acquired IMU data; An information acquisition step of acquiring biometric information of a user; A determination step of determining a state of the user based on the acquired biometric information; and a processing step of changing the IMU data used in the display step according to the determined state of the user.

2. 2. The information processing method according to claim 1, wherein said display step displays light rays by CG in accordance with the IMU data.

3. 2. The information processing method according to claim 1, wherein the display step displays a pointer in accordance with the IMU data.

4. 2. The information processing method according to claim 1, wherein the processing step changes a cutoff frequency of a low-pass filter.

5. 2. The information processing method according to claim 1, wherein the processing step multiplies the IMU data by a predetermined coefficient.

6. 2. The information processing method according to claim 1, wherein the determining step determines a stress state of the user.

7. 2. The information processing method according to claim 1, wherein the determining step determines a state of tension of the user.

8. 2. The information processing method according to claim 1, wherein in said determining step, when a user state is maintained for a predetermined period of time, it is determined that the user state is that state.

9. a UI display step of displaying a user interface for receiving a user operation, and a timer step of measuring time, The information processing method according to claim 1 , wherein the processing step is applied when an operation is attempted in the vicinity of the user interface for a predetermined period of time or longer in the display step.

10. A data acquisition means for acquiring IMU data from an inertial sensor attached to a user; A display means for displaying a user's operation status according to the acquired IMU data; An information acquisition means for acquiring biometric information of a user; A determination means for determining a state of a user based on the acquired biometric information; and processing means for changing the IMU data used by the display means in accordance with the determined state of the user.

11. A program for causing a computer to execute each step of the information processing method according to claim 1.

Images