Operating device, operating system, and operating method

Abstract

The present invention provides an operating device that can appropriately change the method of position tracking operation according to the user's level of fatigue. [Solution] The operating device 1 according to this disclosure comprises a biometric information acquisition unit 12, a fatigue state determination unit 13, and an operation setting change unit 14. The biometric information acquisition unit 12 acquires biometric information of the user performing the operation by position tracking. The fatigue state determination unit 13 determines whether the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit 12. If the fatigue state determination unit 13 determines that the user is in a fatigued state, the operation setting change unit 14 fixes the user's predetermined position tracking.

Description

Technical Field

[0001] The present disclosure relates to an operating device, an operating system, and an operating method.

Background Art

[0002] In virtual reality (VR) technology, in order to move an avatar in a virtual space, a user needs to operate a controller for performing a movement operation or move their own hands and feet. In the method of moving an avatar by moving the user's hands and feet, changes in the user's position can be detected by an acceleration sensor, a gyro sensor, a motion sensor, etc., and the detected changes in position can be reflected in the movement of the avatar. The reflection of such changes in the user's position on the avatar is called position tracking in order to track the user's position.

[0003] Regarding VR technology, Patent Document 1 discloses an information processing device aimed at easily realizing an effective display that can reduce symptoms such as 3D dizziness and fatigue that may occur to an experiencer. The information processing device described in Patent Document 1 includes an index acquisition unit that acquires an index representing symptoms that may occur to an experiencer experiencing a virtual space, a setting acquisition unit that acquires a display setting of an image viewed by the experiencer, and a notification unit that notifies the experiencer of the acquired index and display setting.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, a position tracking operation in a state where the same posture continues for a long time, such as an avatar operation in a standing position for a long time, will give the user a sense of fatigue.

[0006] Therefore, there is a need for the development of technology that can appropriately change the method of position tracking operation according to the user's level of fatigue. However, the technology described in Patent Document 1 does not solve this problem. [Means for solving the problem]

[0007] The operating device according to this disclosure includes a biometric information acquisition unit that acquires biometric information of a user performing an operation by position tracking, a fatigue state determination unit that determines whether or not the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit, and an operation setting change unit that fixes a predetermined position tracking of the user if the fatigue state determination unit determines that the user is in a fatigued state.

[0008] The operating system according to this disclosure comprises an operating target device and an operating device for operating the operating target device, wherein the operating device includes a biometric information acquisition unit for acquiring biometric information of a user performing position tracking operations on the operating target device, a fatigue state determination unit for determining whether the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit, and an operation setting change unit for fixing a predetermined position tracking of the user if the fatigue state determination unit determines that the user is in a fatigued state.

[0009] The operation method described herein involves the operation device acquiring biometric information of a user performing an operation using position tracking, determining whether the user is fatigued based on the acquired biometric information, and, if it is determined that the user is fatigued, fixing the user's predetermined position tracking. [Effects of the Invention]

[0010] According to this disclosure, the method of position tracking operation can be appropriately changed according to the user's level of fatigue. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example configuration of the operating system according to Embodiment 1. [Figure 2] Figure 1 is a schematic diagram showing an example of the operating system. [Figure 3] This is a schematic diagram illustrating an example of the relationship between a user and an avatar using the operating system shown in Figure 1. [Figure 4] This schematic diagram shows another example of the relationship between a user and an avatar using the operating system shown in Figure 1. [Figure 5] This flowchart illustrates an example of the position tracking process performed in the operating system shown in Figure 1. [Figure 6] This schematic diagram shows another example of the relationship between a user and an avatar using the operating system shown in Figure 1. [Figure 7] This flowchart illustrates another example of the position tracking process performed in the operating system shown in Figure 1. [Modes for carrying out the invention]

[0012] The following describes embodiments of the invention, but the invention claimed is not limited to these embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate.

[0013] <Embodiment 1> An example configuration of the operating system according to Embodiment 1 will be described using Figure 1. Figure 1 is a block diagram showing an example configuration of the operating system according to Embodiment 1.

[0014] As shown in Figure 1, the operating system according to this embodiment comprises an operating device 1 and an operating target device 3. The operating device 1 is a device that operates the operating target device 3, and this operation includes position tracking operation, which is an operation by position tracking.

[0015] The operating device 1 and the operation target device 3 are directly connected so as to be communicable by wire or wirelessly, and can transmit and receive information regarding operations. The operating device 1 and the operation target device 3 may be communicable via a wired or wireless network such as a LAN, or may be directly communicable by a cable or short-distance wireless communication or the like.

[0016] Here, although the explanation is made on the premise that the operating device 1 and the operation target device 3 are separate devices, they can be configured as one device. Further, for example, the operation target device 3 may include only some functions of the operating device 1, and the remaining functions may be connected as another device. Hereinafter, configuration examples of the operation target device 3 and the operating device 1 will be described in order.

[0017] (Operation target device 3) The operation target device 3 may include a control unit 30, a storage unit 36, a communication unit 37, and a display unit 38.

[0018] Regardless of its type, the operation target device 3 may be any device capable of position tracking operations. The operation target device 3 may be, for example, a device that provides a music live in a virtual space, or a game device that provides a game in a virtual space, or a communication device that communicates with other users in a virtual space. Of course, even when the operation target device 3 is a game device, the operation target device 3 can have a function as a communication device. In the operation target device 3, it is possible to move an avatar imitating a user in a virtual space.

[0019] The operation target device 3 can be configured to include a portable or installed computer such as an HMD (Head Mounted Display), a mobile phone, a smartphone, a mobile PC (Personal Computer), or a PC. As exemplified by the computer, the operation target device 3 may be an information processing device.

[0020] The device to be operated 3 can also be constructed as a system in which functions are distributed among multiple devices. For example, if the device to be operated 3 is configured to include an HMD equipped with a display unit 38, the device to be operated 3 may include a portable or stationary computer connected to the HMD by wire or wireless connection and on which application programs such as games are installed. Alternatively, for example, the device to be operated 3 may include a computer installed on the user's side and a server device connected to that computer via a network that provides the computer with application programs such as games that provide a virtual space to the user.

[0021] The control unit 30 controls the entire target device 3. The control unit 30 may be implemented by, for example, a processor such as a CPU (Central Processing Unit), working memory, and a non-volatile storage device that stores a control program. This program may include a program that, when executed by the processor, realizes the functions of the target device 3. This program may include application programs such as games. The control unit 30 may also be implemented using a programmable integrated circuit, such as an FPGA (field-programmable gate array) or a microcomputer.

[0022] The control unit 30 may include a main control unit 31, an input control unit 32, and a display control unit 33, which are functional blocks realized in the target device 3 by the operation of a program, etc., and are described later. The main control unit 31 may include an application program such as a game, and realizes various functions of the target device 3 while controlling the input control unit 32 and the display control unit 33.

[0023] The memory unit 36 ​​is a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive) that stores various types of information used by the device to be operated 3. As one type of information to be stored, the memory unit 36 ​​stores, for example, user setting information in application programs such as games. The input control unit 32 reads the information stored in the memory unit 36 ​​and updates the information stored in the memory unit 36.

[0024] The communication unit 37 communicates with external devices such as the operating device 1. The communication unit 37 is, for example, a communication unit that communicates in accordance with wireless communication standards such as Wi-Fi (registered trademark; the same applies hereinafter), or a communication unit that communicates in accordance with short-range wireless communication standards such as Bluetooth (registered trademark; the same applies hereinafter). The communication unit 37 may also be equipped with a communication unit for any mobile communication system in order to communicate with external devices other than the operating device 1.

[0025] The input control unit 32 sends and receives information with external devices such as the operating device 1 via the communication unit 37. When the input control unit 32 receives information indicating the operation content from the operating device 1, it passes that operation content to the main control unit 31. The main control unit 31 controls the display control unit 33 and other components according to this operation content.

[0026] The display unit 38 may be composed of a display device such as a liquid crystal panel or an organic electroluminescent panel. The display control unit 33 controls the display on the display unit 38 and changes the image representing the virtual space displayed on the display unit 38. For example, the display control unit 33 changes the image to be displayed on the display unit 38 according to the control from the main control unit 31 that corresponds to the operation content indicated by the information received from the operating device 1.

[0027] (Operating device 1) The operating device 1 detects when the user is fatigued based on the user's biometric information and, in response to the fatigued state, changes the settings to allow the user to continue their virtual experience as if they were healthy. For this purpose, the operating device 1 may include a control unit 10, a storage unit 16, a communication unit 17, an input unit 18, a motion sensor 19, and a biosensor 20. The operating device 1 can be any device capable of accepting position tracking operations, regardless of its type or configuration.

[0028] The control unit 10 controls the entire operating device 1. The control unit 10 may be implemented by, for example, a processor such as a CPU, working memory, and a non-volatile storage device that stores a control program. This program may include a program that, when executed by the processor, realizes the functions of the operating device 1. Alternatively, the control unit 10 may be implemented using an integrated circuit that can be programmed by the user, such as an FPGA or a microcomputer.

[0029] The control unit 10 may include a tracking processing unit 11, a biological information acquisition unit 12, a fatigue state determination unit 13, and an operation setting change unit 14, which are functional blocks realized by the operation of a program in the operating device 1. The control unit 10 may also include a posture change determination unit 15, and a configuration including the posture change determination unit 15 will be described later as Embodiment 2.

[0030] The storage unit 16 is a storage device such as an SSD or HDD that stores various types of information used by the operating device 1. As one type of information to be stored, the storage unit 16 stores, for example, user setting information for the operating device 1.

[0031] The communication unit 17 communicates with the device to be operated 3. The communication unit 17 may be a communication unit such as the one exemplified as the communication unit 37.

[0032] The input unit 18 is a part that accepts user input and can also be called the operation unit. The input unit 18 is preferably provided to supplement the movement operation of the avatar by the motion sensor 19, but the operation device 1 may be configured without an input unit 18. The input unit 18 may consist of at least one of a physical key and a software key, or it may consist of a separately provided voice input unit and voice recognition unit, or it may consist of both. The input unit 18 may also include a touch sensor incorporated into a separately provided display unit (not shown). In that case, the operation device 1 will include a display unit and a touch panel as an example of the input unit 18.

[0033] The motion sensor 19 is one or more sensors that detect the user's movements and can consist of, for example, a gyroscope and an accelerometer. The motion sensor 19 can be, for example, a 6DoF (Degree of Freedom) sensor, but it is sufficient that it can detect changes in the position of one or more parts of the user's body, that is, the movement of one or more parts of the user. The motion sensor 19 may also include a motion capture camera installed at a location separate from the main body of the operating device 1.

[0034] Furthermore, if the device to be operated 3 includes an HMD equipped with a display unit 38, at least a portion of the motion sensor 19 of the operating device 1 may be mounted on the HMD as a sensor for detecting head movement. In this case, the motion sensor 19 may include not only position tracking sensors, but also head tracking sensors, eye tracking sensors, etc. Note that some or all of the head tracking sensors may be position tracking sensors.

[0035] The motion sensor 19 transmits information indicating the detected user movement to the tracking processing unit 11. Based on the user movement detected by the motion sensor 19, the tracking processing unit 11 determines the content of the position tracking operation and outputs information indicating the content of that operation to the target device 3 via the communication unit 17. The tracking processing unit 11, or the tracking processing unit 11 and the communication unit 17, can be referred to as a VR controller, for example, because they instruct the user's movements to be reflected in the virtual space.

[0036] The input unit 18 can consist of, for example, a key for turning the power on and off, but it can also include a key that accepts user movement and direction of movement instructions instead of detecting user movement with the motion sensor 19. In that case, the control unit 10 should transmit these instructions to the device to be operated 3 via the communication unit 17.

[0037] The biosensor 20 is a group of sensors consisting of one or more types of sensors that acquire the user's biometric information, or a device equipped with such a group of sensors. For example, the biosensor 20 can be mounted on wearable devices such as bracelets, rings, glasses, or in-ear devices. Examples of such wearable devices include smartwatches, smart rings, smart glasses, and smart earphones. Furthermore, if the device to be operated 3 includes an HMD equipped with a display unit 38, at least a portion of the biosensor 20 of the operating device 1 can be mounted on the HMD.

[0038] Furthermore, if at least a portion of the motion sensor 19 or biosensor 20 is mounted in a separate housing from the main body of the operating device 1, the separate housing may be equipped with a communication unit that communicates wirelessly or via wired connection with the main body in order to transmit the detection results to the main body.

[0039] The biosensor 20 detects biometric information that allows for the estimation of the user's fatigue level or whether the user is fatigued, based on its detection results or changes in those results over time. This biometric information is also referred to as vital information. The biometric information may be one or more of various pieces of information, such as pulse waves showing fluctuations in pulse rate, body temperature, blood pressure, respiratory rate, electroencephalogram (EEG), and cerebral blood flow. Cerebral blood flow can be detected, for example, by an optical sensor mounted on an in-ear wearable device worn on the ear, or by an optical sensor mounted near the ear of an HMD (Head-Mounted Display). The biosensor 20 may also include, for example, an audio input unit consisting of a microphone for inputting the user's voice, or a camera for capturing images of the user's face.

[0040] The biometric information acquisition unit 12 acquires biometric information from the biosensor 20 and passes it to the fatigue state determination unit 13. In particular, the biometric information acquisition unit 12 acquires biometric information of the user performing position tracking operations on the target device 3 and passes it to the fatigue state determination unit 13. The biometric information acquisition unit 12 may generate biometric information in a format to be used for determination by the fatigue state determination unit 13 by, for example, extracting characteristic quantities of the biometric information from the acquired biometric information, and then pass it to the fatigue state determination unit 13.

[0041] Feature quantities of biometric information can be values ​​that represent the characteristics of the waveform of the pulse wave, for example, if the acquired biometric information is pulse wave information, and similarly for other types of biometric information, they can be values ​​that represent the characteristics of that biometric information. If the biosensor 20 is equipped with an audio input unit, the biometric information acquisition unit 12 may extract feature quantities or changes in feature quantities such as the intonation and volume of the user's voice from the audio data acquired by the audio input unit. If the biosensor 20 is equipped with a camera, the biometric information acquisition unit 12 may extract feature quantities or changes in feature quantities such as the user's facial expression, gaze, and pupil size from the image data captured by the camera. Feature quantities of facial expression can refer to features such as the angle of the corners of the mouth and the corners of the eyes.

[0042] Furthermore, in order to extract features from biological information, the biological information acquisition unit 12 may be equipped with a pre-trained model that has been machine-learned to take the biological information received from the biological sensor 20 as input and output features. The algorithm and other aspects of this pre-trained model are not limited.

[0043] The fatigue state determination unit 13 determines whether the user is fatigued or not based on the biometric information received from the biometric information acquisition unit 12. For example, the fatigue state determination unit 13 calculates the user's fatigue level based on the biometric information received from the biometric information acquisition unit 12, and determines whether the user is fatigued or not based on whether that fatigue level is above a threshold. The fatigue state determination unit 13 passes the determination result to the operation setting change unit 14.

[0044] The fatigue state determination unit 13, for example, if electroencephalogram (EEG) information indicates a state of sympathetic nervous system dominance, may determine the psychological state to be excited if the EEG is β waves, while determining it to be a state of concentration if the EEG is γ waves. The fatigue state determination unit 13 then determines the psychological state to be stressed if the duration of the excited state is longer than a first predetermined period, and estimates that the user is in a fatigued state if the stressed state continues for a second predetermined period longer than the first predetermined period. Although only EEG information has been used as an example, the fatigue state determination unit 13 can perform similar determinations from other biological information based on the relationship between that biological information and the fatigue state. For example, the fatigue state determination unit 13 can determine that the user is in a fatigued state if the cerebral blood flow is lower than a predetermined amount.

[0045] To calculate the degree of fatigue or determine whether or not a person is fatigued, the fatigue state determination unit 13 may include a trained model that has been trained to take biological information as input and output the degree of fatigue or whether or not a person is fatigued. The algorithm of this trained model is not limited.

[0046] The operation setting change unit 14 receives the determination result from the fatigue state determination unit 13. If the operation setting change unit 14 determines that the user is fatigued, it fixes the user's predetermined position tracking. In other words, if the operation setting change unit 14 determines that the user is fatigued, it changes the operation settings to fix the user's predetermined position tracking. Information indicating the operation settings can be stored in the storage unit 16 as part of the user setting information described above.

[0047] The specified position tracking can be, for example, position tracking in the height direction, i.e., vertical position tracking, but is not limited to this; it may also be position tracking in the left-right direction, etc.

[0048] Fixing vertical position tracking means that changes in height are not reflected in the user's position tracking. Not reflecting vertical changes is beneficial in virtual spaces for events such as music concerts where users stand and sway their bodies. Fixing horizontal position tracking means that changes in the horizontal direction are not reflected in the user's position tracking. Not reflecting horizontal changes is beneficial in virtual spaces for events such as graduation ceremonies where the user's side-to-side swaying is noticeable.

[0049] Furthermore, the predetermined position tracking may fix the target body part, such as the upper body in the case of vertical position tracking. Also, even when fixing vertical position tracking, only the center position of that body part may be fixed. This allows, for example, even when fixing the vertical position tracking of the head, to track the user's nodding motion and reflect it in the avatar.

[0050] Furthermore, the predetermined position tracking can be varied depending on the type of space represented in the virtual space, such as a live music venue, bedroom, or plateau, that is, depending on the conditions of the provided virtual space. Also, the process of fixing the predetermined position tracking may be set to be either performed or not performed for each user, or a threshold for determining fatigue level may be set for each user. The contents of such settings can be stored in the storage unit 16 as user setting information for each user.

[0051] An example of the application of such an operating system and its effects will be briefly explained. As an example, consider a case where a user is participating in a VR live performance. In this case, if the user stands for a long time participating in the VR live performance, fatigue will accumulate in the user's legs. At that time, the biometric information acquisition unit 12 acquires biometric information from one or more biosensors 20, such as a microphone for acquiring sound, a wearable device for measuring brain waves, and a camera for capturing facial expressions, for the user participating in the VR live performance. Then, the fatigue state determination unit 13 determines from this biometric information whether or not the user is tired. The fatigue state determination unit 13 will determine that the user is in a fatigued state if the user has been standing for a long time to the point where fatigue has accumulated in the user's legs.

[0052] Thus, when it is determined that the user is fatigued, the operation setting change unit 14 changes the operation settings to disconnect the position tracking of the user's feet and body height, thereby fixing the position tracking of the user's feet and body height. Disconnecting position tracking here means that the tracking processing unit 11 does not reflect the content of the position tracking operation determined based on the information indicating the user's movement detected by the motion sensor 19 onto the avatar in the virtual space. In other words, when the operation settings are changed as described above, the tracking processing unit 11, or the tracking processing unit 11 and the communication unit 17, do not reflect the change in foot and body height detected by the motion sensor 19 onto the avatar in the virtual space. Fixing position tracking means not changing the coordinates of the position at the time it is determined that the user is fatigued, or at the time it disconnects position tracking. For this reason, the tracking processing unit 11 may, for example, store the coordinates of the foot and body height at the time it is determined that the user is fatigued, or at the time it disconnects position tracking, in the storage unit 16, and continue to use those coordinates without updating them thereafter.

[0053] Specifically, if the fatigue level measurement indicates that the user is fatigued, the operation setting change unit 14 changes the method of controlling the avatar in the VR live space so that the height of the feet and body controlled by the gyro sensor or HMD position tracking is not reflected in the avatar in the virtual space. In such an operation system, for example, by disconnecting the position tracking of a fatigued user, it is possible to suppress the shaking of the virtual space image or avatar displayed on the display unit 38 in response to the user being unsteady or swaying.

[0054] Next, using Figures 2 to 4, we will describe an example of an operating system in which the main part of the operating device 1 and the operating target device 3 are integrated. Of course, the operating system shown in Figure 2 is merely one example, and other configurations can also be applied. Figure 2 is a schematic diagram showing an example of the operating system in Figure 1. Figure 3 is a schematic diagram showing an example of the relationship between a user and an avatar using the operating system in Figure 1. Figure 4 is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system in Figure 1.

[0055] The operating system shown in Figure 2 comprises an HMD2 worn on the head of user U, a sensor 19b which is part of a motion sensor 19 installed at a distance from user U, and a sensor 20b which is part of a biosensor 20 worn on user U's arm. Sensor 19b is a motion capture sensor that detects user U's movements as changes in the position of various parts of user U, such as those shown by the black circles in Figure 2. In Figure 2, an example is shown where sensor 19b is installed on a table, but the installation method is not limited to hanging from the ceiling, etc. Sensor 20b is a wearable device that measures user U's pulse, for example.

[0056] The HMD2 is equipped with the target device 3 and the main part of the operating device 1. The main part of the operating device 1 mounted on the HMD2 includes sensor 19a, which is part of the motion sensor 19, and sensor 20a, which is part of the biosensor 20. Sensor 19a is, for example, a gyroscope and an accelerometer. Sensor 20a is, for example, an optical sensor that detects cerebral blood flow.

[0057] In Figure 3, User Ua represents User U in a standing position, and for convenience, Avatar AVa is shown alongside User Ua. Avatar AVa is a virtual avatar in space resulting from tracking User Ua with the motion sensor 19. User Ua and Avatar AVa coincide in the height direction at the part of the body targeted for position tracking, indicated by the black circles.

[0058] Figure 4 shows a scenario where user U has accumulated fatigue and is sitting in chair CH. In Figure 4, user U in the seated position is represented as user Ub, and for convenience, avatar AVb is shown next to user Ub.

[0059] Avatar AVb is an avatar in virtual space resulting from tracking the user Ub with the motion sensor 19, and is the avatar when settings are changed in the operation setting change unit 14 in this embodiment. In Figure 4, it can be seen that the user Ub and avatar AVb are misaligned in the height direction in at least a part of the body targeted for position tracking, indicated by the black circles, specifically in the upper body including the head, hands, and torso.

[0060] This is because a setting change was made that fixed the vertical position tracking of user Ub. More specifically, the operation setting change unit 14 fixed avatar AVb in the same standing position as avatar AVa corresponding to user Ua in a standing position, and did not reflect the change in upper body height when user Ua became user Ub. In this case, position tracking other than vertical positioning continues, and head tracking and eye tracking, if performed, also continue.

[0061] On the other hand, if this embodiment is not applied and no settings are changed in the operation setting change unit 14, an avatar in a seated position with the same posture as the user Ub will be displayed (not shown in the figure).

[0062] (Processing in the operating system) Next, using Figure 5, we will explain an example of the position tracking process performed by the operating device 1 of the operating system shown in Figure 1. Here again, we will use an operating system including the HMD2 as an example, as shown in Figure 2. Figure 5 is a flowchart illustrating this example of position tracking process.

[0063] First, as shown in Figure 2, the user U puts on the HMD2 and sensor 20b and stands within the detection range of sensor 19b, so that the device to be operated 3 can be operated by the operating device 1. Next, the user U, for example, when standing, inputs a calibration execution operation from the input unit 18. The control unit 10 of the operating device 1 receives this calibration execution operation and performs the calibration (step S11).

[0064] Calibration involves initial setup, where the user U assumes the same pose as the displayed avatar, thereby synchronizing the user U's movements with those of the avatar. Performing initial setup in multiple postures is advantageous in improving the accuracy of reflecting the user U's movements to the avatar. Therefore, calibration should be performed in various scenarios where the user U assumes multiple postures, such as standing and sitting.

[0065] In step S11, specifically, the tracking processing unit 11 instructs the motion sensor 19, which consists of sensors 19a and 19b, to perform detection, and registers the initial values ​​such as the user's initial position and initial angle detected by the motion sensor 19 in the storage unit 16 as part of the user setting information. If eye tracking is performed, the tracking processing unit 11 may also register the initial value of the user's gaze detected by the eye tracking sensor in the storage unit 16 as part of the user setting information. In addition, the tracking processing unit 11 may perform room scale measurement to measure the floor position, room size, etc., and register these in the storage unit 16 as part of the user setting information.

[0066] Furthermore, calibration may include, if necessary, superimposing an avatar corresponding to user U onto an image representing the virtual space on the display unit 38 of the device to be operated 3, and registering the avatar's position and size as part of the user setting information in the storage unit 16.

[0067] Furthermore, calibration may include calibration of the biosensor 20. Specifically, when a calibration operation is received, the bioinformation acquisition unit 12 instructs the biosensor 20 to detect bioinformation, and obtains initial values ​​of the bioinformation by extracting its features from the biosensor 20. The bioinformation acquisition unit 12 then registers these initial values ​​as part of the user setting information in the storage unit 16.

[0068] Once the calibration in step S11 is complete, the tracking processing unit 11 starts accepting tracking operations, including position tracking operations, and begins the tracking process (step S12). As a result, the display unit 38 of the target device 3 displays an avatar on the image representing the virtual space, reflecting the user's tracking operations relative to the user U's initial values.

[0069] As step S12 begins, the biometric information acquisition unit 12 acquires biometric information of user U performing the position tracking operation from the biosensor 20 (step S13). This acquisition only needs to be performed while the tracking operation continues, but it does not need to be performed after step S15, which will be described later.

[0070] Next, the fatigue state determination unit 13 calculates the fatigue level of user U based on the biological information acquired by the biological information acquisition unit 12, and determines whether user U is in a fatigued state based on whether the fatigue level is above a predetermined level (step S14). If the result in step S14 is NO, the process returns to step S13, and the acceptance of tracking operations and acquisition of biological information continues.

[0071] On the other hand, if the answer in step S14 is YES, that is, if it is determined that user U is in a fatigued state, the operation setting change unit 14 fixes predetermined position tracking, such as vertical position tracking of user U (step S15).

[0072] As a result, as illustrated in Figure 4, for example, even if user U is sitting in a chair, if they are fatigued, the vertical tracking of user U will be fixed, and the avatar will be generated and displayed based on other tracking.

[0073] Even after processing in step S15, tracking, including position tracking, continues (step S16). However, with regard to position tracking, position tracking other than the predetermined position tracking will continue.

[0074] Next, the control unit 10 determines whether or not it has received a termination operation from the input unit 18 (step S17), and terminates the process if it has received one. If the result in step S17 is NO, it is advisable to continue tracking, including the position tracking in step S16.

[0075] Furthermore, although not shown in Figure 5, the acquisition of biological information in step S13 may be continued after step S15, and the operation setting change unit 14 may release the predetermined position tracking lock when the fatigue state determination unit 13 determines that user U is no longer fatigued.

[0076] As described above, according to this embodiment, the method of position tracking operation can be appropriately changed depending on whether the user is energetic or tired, that is, according to the user's level of fatigue. As a result, according to this embodiment, for example, it is possible to suppress the shaking of the virtual space image or avatar displayed on the display unit 38 in response to the user being unsteady due to fatigue.

[0077] <Embodiment 2> The operating system according to Embodiment 2 will be explained using Figures 6 and 7, focusing on the differences from Embodiment 1, but various examples described in Embodiment 1 can be applied. Figure 6 is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system of Figure 1.

[0078] In this embodiment, the control unit 10 includes a posture change determination unit 15. The posture change determination unit 15 determines whether or not the user's posture has changed based on the biological information acquired by the biological information acquisition unit 12. Examples of biological information used for this determination include blood pressure, heart rate, respiratory rate, and cerebral blood flow. Systolic and diastolic blood pressure can be used for this determination because they are slightly lower in a sitting position than in a standing position. Heart rate can be used for this determination because it may temporarily decrease when changing from a standing to a sitting position. Respiration is more stable in a sitting position than in a standing position, and the respiratory rate is lower in a sitting position than in a standing position, so these can be used for this determination. Cerebral blood flow can be used for this determination because it may temporarily increase when changing from a standing to a sitting position.

[0079] This change in posture can be defined by a predetermined state change. A predetermined state change may refer to, for example, a change from standing to sitting, a change from standing with both feet together to standing with one foot apart, or a change from standing to lying on one's back, on one's stomach, or on one's side. Other predetermined state changes may refer to, for example, changes between lying on one's stomach, on one's back, or on one's side, or a change from sitting to lying on one's back, on one's stomach, or on one's side. Each state can be predetermined based on the positional relationships between the coordinates of the user's various body parts. Even for changes other than between standing and sitting, biological information such as blood pressure, heart rate, respiratory rate, and cerebral blood flow may change, and if the trends of these changes are learned in advance, the state change can be determined from the changes in that biological information.

[0080] Furthermore, the posture change determination unit 15 may calculate and acquire the posture of the user wearing the HMD2, including the head position, head orientation, and movement, based on detection information from acceleration sensors, angular velocity sensors, etc., mounted on the HMD2. One posture calculation method is a posture estimation method that uses an inertial measurement unit (IMU) to estimate the posture. In this method, a gyroscope, accelerometer, and magnetometer are combined to acquire the three-dimensional coordinates of each part of the user and estimate the posture. Another posture calculation method is to use a depth sensor such as Kinect® to acquire the user's joint positions in real time and estimate the posture. Alternatively, the user's posture may be determined by performing image analysis using an external camera.

[0081] Thus, the posture change determination unit 15 can also calculate posture based on detection information from sensors other than the biosensor 20 and determine changes in posture based on the calculated results. Furthermore, the posture change determination unit 15 may determine changes in posture based on detection information from both the biosensor 20 and sensors other than the biosensor 20. For example, a trained model may be created that combines biometric information such as fluctuations in heart rate and respiratory rate, or changes in body temperature, with detection information from other sensors, and is machine-learned to take biometric information and detection information as input and output changes in posture. The posture change determination unit 15 may then obtain changes in posture based on this trained model. The algorithm of this trained model is not specified.

[0082] In this embodiment, the operation setting change unit 14 fixes the predetermined position tracking when the fatigue state determination unit 13 determines that the user is in a fatigued state and the posture change determination unit 15 determines that the user's posture has changed.

[0083] In the following, as with the example in Embodiment 1, we will provide an example where the predetermined position tracking is vertical position tracking, and an example where the change in posture is a change from a standing position to a sitting position.

[0084] If the fatigue state determination unit 13 determines that the user is fatigued and the posture change determination unit 15 determines that the user's posture has changed from standing to sitting, the operation setting change unit 14 performs the following fixation. Specifically, in this case, the operation setting change unit 14 fixes the vertical position tracking of the user's feet and body by changing the operation settings to disconnect the vertical position tracking of the user's feet and body. The coordinates used to fix the position tracking are the coordinates at the time the user was determined to be in a standing state.

[0085] This fixed positioning disconnects vertical position tracking, so vertical movements of the user's body are not reflected in the avatar. As a result, even if the user becomes tired and sits down, the avatar can maintain an upright position.

[0086] In this case, if position tracking is completely disabled, no body movements will be reflected in the avatar. Therefore, the information that the character is sitting is ignored, and other movements such as arm swings or body sway are reflected instead. In other words, in this embodiment, in the tracking control including position tracking, other movements are reflected without reflecting that the character is sitting.

[0087] More specifically, in this embodiment, when the user sits down due to fatigue, the height changes of each tracking point are not reflected, so the avatar's upper body maintains the same height as when standing. As a result, in this embodiment, the user's viewpoint height is maintained even when sitting, thus maintaining the field of view as when standing. Therefore, in this embodiment, the avatar can be operated with a standing gaze, allowing the user to continue enjoying VR live performances and other content.

[0088] To elaborate, if position tracking were functioning normally when a user changes from a standing to a sitting position, the user's eye level would also decrease. Therefore, in this embodiment, the eye level is maintained without change by disabling vertical position tracking. This maintenance is particularly beneficial in situations such as VR live performances, where it is necessary to maintain the user's gaze from a standing position even when they are sitting.

[0089] For example, it is assumed that users participating in a VR live event are standing for extended periods, such as 30 minutes to an hour or more. In this case, if a user who has been standing for a long time changes their posture to sitting or lying down, it is thought that the user wanted to maintain the field of view from their standing position, but changed their posture due to fatigue. Therefore, in this embodiment, the standing position for a long time is defined as the first posture, and when it is determined that the posture has changed from the first posture, it is determined whether the user is tired or not, and if the user is tired, the height of the viewpoint is fixed. This prevents malfunctions such as fixing the height of the viewpoint when it is desired that the user's sitting or lying-down movements be reflected in the avatar.

[0090] Furthermore, if the fatigue state determination unit 13 determines that the user is fatigued, the operation setting change unit 14 may reset the user's post-change posture as the initial value of the user's pre-change posture. The user's post-change posture refers to the posture in which the user has settled after changing their posture, that is, the posture that has been maintained for a predetermined period of time after the change. Also, resetting the user's posture as the initial value means resetting the initial value of the avatar's posture corresponding to the user, that is, performing position tracking calibration again. As a result, the difference between the value detected by the motion sensor 19 and the redefined initial value will be reflected in the avatar.

[0091] During the initial calibration, the user's movements are synchronized with those of the avatar by having the user assume the same pose. This means that even if the initial calibration is performed in various postures, the user's standing position is registered as the initial value for the avatar's upright position. Therefore, after the initial calibration, the user's standing position is usually linked to the avatar's standing position. Consequently, if a user changes to a seated position and tries to manipulate an avatar in a standing position using motion tracking, the resulting movement may be unintended by the user.

[0092] In contrast, as in this embodiment, if the user's posture after a change, such as sitting, is reset as the initial value of the user's posture before the change, the user's seated state and the avatar's standing state can be linked. In other words, such a reset means recalibrating the system by treating the user's seated state as if the user were standing. As a result, even if the user changes from a standing state to a seated state, if the user is fatigued, they can still operate the avatar in a standing state.

[0093] Let's explain this using Figure 6. In Figure 6, as in Figure 4, fatigue has accumulated in User U, and User U is sitting in chair CH. However, in this seated position, User U has extended their legs and put their feet forward. In Figure 6, User U in this seated position with their legs extended is shown as User Uc, and for convenience, Avatar AVc is shown next to User Uc.

[0094] Avatar AVc is an avatar in the virtual space resulting from tracking the user Uc with the motion sensor 19, and is the avatar when settings are changed in the operation setting change unit 14 in this embodiment. As in Figure 6, similar to Figure 4, it can be seen that the user Ub and avatar AVb are misaligned in the height direction in the head, hands, and torso, which are the targets of position tracking indicated by the black circles.

[0095] However, in Figure 6, when User U steps forward with their leg, as in User Uc, the avatar AVc performs a kicking motion with the hips as its axis. To enable the reflection of such movements, during recalibration, it is recommended to set up the avatar to associate, for example, the action of extending the lower leg while seated with the action of stepping forward with the leg. When this action is reflected in the avatar, the action of User U stepping forward with their leg, as in User Uc, will be reflected in the avatar AVc as a kicking motion with the hips as its axis.

[0096] Furthermore, through recalibration, the user Uc and the avatar AVc may be made the same in the height direction of the feet, as shown in Figure 6, or they may be made different according to the ratio between the height of the chair CH and the length of the user U's legs. If they are made different according to the ratio, it means that the avatar AVc can only raise its feet up to the height of the chair CH.

[0097] Figure 7 is a flowchart illustrating another example of the position tracking process performed in the operating system shown in Figure 1.

[0098] First, the same process as in steps S11 to S14 in Figure 5 is executed (steps S31 to S34). If the answer in step S34 is NO, the process returns to step S33, and the acceptance of tracking operations and acquisition of biological information continues.

[0099] On the other hand, if the answer in step S34 is YES, that is, if it is determined that user U is in a fatigued state, the posture change determination unit 15 determines whether or not the user's posture has changed (step S35). As described above, the determination in step S35 can be, for example, whether or not user U has changed from a standing position to a sitting position. If the answer in step S35 is NO, the system returns to step S33 and continues to accept tracking operations and acquire biometric information.

[0100] On the other hand, if the answer in step S35 is YES, that is, if it is determined that user U has changed from a standing position to a sitting position, the operation setting change unit 14 fixes predetermined position tracking, such as vertical position tracking of user U (step S36). The process in step S36 is the same as the process in step S15 in Figure 5.

[0101] Next, the control unit 10 of the operating device 1 performs calibration so that the avatar of the pre-change posture can be represented in the changed posture (step S37). The calibration in step S37 is a recalibration, but the calibration method itself is as described in step S11 of Figure 5. However, the recalibration differs from the initial calibration in that it is not triggered by a calibration operation from the user U, and it is performed so that the avatar of the pre-change posture can be represented in the changed posture. The calibration in step S37 may or may not include calibration of the biosensor 20.

[0102] After the processing in step S37, tracking, including position tracking, continues (step S38), similar to step S16 in Figure 5. However, in step S38, tracking is performed based on the user setting information after the calibration is performed again in step S37. Of course, in step S38 as well, position tracking other than the predetermined position tracking will continue.

[0103] Next, similar to step S17 in Figure 5, the control unit 10 determines whether or not it has received a termination operation from the input unit 18 (step S39), and terminates the process if it has received one. If the result in step S39 is NO, it is advisable to continue tracking, including the position tracking in step S38.

[0104] Furthermore, although not shown in Figure 7, the acquisition of biological information in step S33 may be continued after steps S36 and S37. Then, when user U is no longer fatigued or returns from a seated position to a standing position, the operation setting change unit 14 may release the fixed position tracking.

[0105] According to this embodiment, by determining changes in the user's posture and fixing a predetermined position tracking, it becomes unnecessary to fix the predetermined position tracking when there is no change in posture, such as when the user is tired but not to the point of changing their posture. Furthermore, according to this embodiment, by adopting a configuration that resets the initial values, the user can operate the avatar in the standing position even when sitting, and can operate the avatar in the state before the posture change even after changing their posture.

[0106] Note that while Figure 7 shows a recalibration process after a change in posture, this is not the only option. For example, the operation setting change unit 14 may record the avatar's posture before the change and the position tracking results for the posture after the change in a database format in the storage unit 16. This allows the tracking processing unit 11 to refer to this database and perform position tracking operations on the avatar as if the user's posture had not changed, even after the user has changed their posture. For example, the operation setting change unit 14 may store the movements of a seated state and an upright state in association, allowing the user to operate the upright avatar even when seated.

[0107] Furthermore, a process that produces a similar effect to this example of resetting initial values ​​can also be applied to Embodiment 1, which does not involve determining changes in the user's posture or fixing a predetermined position tracking based on the determination result.

[0108] In other words, in Embodiment 1, if the fatigue state determination unit 13 determines that the user is fatigued, the operation setting change unit 14 may set the operation so that the user's avatar maintains the posture of the user before fatigue, while corresponding to the user's posture after fatigue. The user's avatar is the target of operation by the user's position tracking. For example, the operation setting change unit 14 may record in a database format in the storage unit 16 the avatar's posture before the user became fatigued and the position tracking results of the user's posture after fatigue. This allows the tracking processing unit 11 to refer to this database and perform position tracking operations on the avatar as if the user were not fatigued, even after the user has become fatigued.

[0109] <Embodiment 3> In embodiments 1 and 2, the operation setting change unit 14 fixed the user's predetermined position tracking when it determined that the user was fatigued. Such fixing is useful, for example, in VR live performances, but there are also virtual spaces where it is desirable to represent standing and sitting states with avatars.

[0110] Therefore, in this embodiment, the operation setting change unit 14 performs such fixing only when the determination result of the posture change determination unit 15 indicates that the user has remained in an upright position for a predetermined period of time, or when the same state has continued for a predetermined period of time. In other words, in this embodiment, the operation setting change unit 14 fixes a predetermined position tracking when the fatigue state determination unit 13 determines that the user is in a fatigued state and the posture change determination unit 15 determines that the user's posture has changed after remaining unchanged for a predetermined period of time.

[0111] According to this embodiment, except in cases where the user's posture has not changed for a long period of time, the avatar can represent any posture, such as standing or sitting, even when the user is tired.

[0112] Furthermore, in this embodiment, the predetermined position tracking is fixed when the user becomes fatigued, but in a further application example, such fixing may be performed without considering the user's fatigue. That is, in this application example, the fatigue state determination unit 13 and the biosensor 20 do not need to be provided in the operating device 1. In this application example, the operation setting change unit 14 fixes the predetermined position tracking when the posture change determination unit 15 determines that the user's posture has changed after remaining unchanged for a predetermined period of time.

[0113] As shown in this application example, by determining whether or not to perform a process to fix a predetermined position tracking based solely on the time related to changes in posture, without considering the user's fatigue level, it is possible to fix the desired position tracking regardless of the user's fatigue level.

[0114] <Alternative examples, etc.> Some or all of the processing in the operating device 1, HMD 2, or the device to be operated 3 described above can be implemented as a computer program. The program described above includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments when loaded into a computer. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0115] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

[0116] For example, the biometric information mentioned above can be other information than that exemplified, and the position tracking method and the type of device being operated are not limited to those exemplified. Furthermore, the above embodiment was described on the premise that appropriate modifications to the position tracking operation method are applied to VR technology. However, such appropriate modifications to the position tracking operation method may also be applied to mixed reality (MR) technology or augmented reality (AR) technology, which merges real space and virtual space in real time. [Explanation of Symbols]

[0117] 1 Operating device 2 HMD 3. Device to be operated on 10, 30 Control Unit 11 Tracking Processing Unit 12. Biological Information Acquisition Unit 13. Fatigue state determination unit 14 Operation Settings Change Section 15 Posture change determination unit 16, 36 storage section 17, 37 Communications Department 18 Input section 19 Motion Sensors 19a, 19b sensors 20 Biosensors 20a, 20b sensors 31 Main Control Unit 32 Input Control Unit 33 Display Control Unit 38 Display section U, Ua, Ub, Uc users AVa, AVb, AVC (Avatar)

Claims

1. A biometric information acquisition unit that acquires biometric information of a user performing an operation using position tracking, A fatigue state determination unit determines whether or not the user is in a fatigued state based on the biological information acquired by the biological information acquisition unit, If the fatigue state determination unit determines that the user is in a fatigued state, the operation setting change unit fixes the user's predetermined position tracking, An operating device equipped with the following features.

2. The system further includes a posture change determination unit that determines whether or not the user's posture has changed based on the biological information acquired by the biological information acquisition unit. The operation setting change unit fixes the predetermined position tracking when the fatigue state determination unit determines that the user is in a fatigued state and the posture change determination unit determines that the user's posture has changed. The operating device according to claim 1.

3. When the fatigue state determination unit determines that the user is in a fatigued state, the operation setting change unit sets the operation settings so that the avatar, which is the target of operation by the user's position tracking, maintains the posture of the user before fatigue, while corresponding to the posture of the user after fatigue. The operating device according to claim 1 or 2.

4. The system comprises a device to be operated and an operating device for operating the device to be operated. The aforementioned operating device is A biometric information acquisition unit acquires biometric information of a user performing position tracking operations on the aforementioned target device, A fatigue state determination unit determines whether or not the user is in a fatigued state based on the biological information acquired by the biological information acquisition unit, If the fatigue state determination unit determines that the user is in a fatigued state, the operation setting change unit fixes the user's predetermined position tracking, Equipped with, Operating system.

5. By acquiring biometric information of the user performing the operation using position tracking, Based on the acquired biometric information, it is determined whether the user is in a fatigued state. If it is determined that the user is in a fatigued state, the user's predetermined position tracking is fixed. A method of operation in which a processing device performs the operation.

Images