Operation device, operation system, and operation method

Abstract

The present invention provides an operation device capable of appropriately changing a method of position tracking operations according to a sensation of fatigue of a user. An operation device (1) according to the present disclosure includes a biometric information acquisition unit (12), a fatigue state determination unit (13), and an operation settings changing unit (14). The biometric information acquisition unit (12) acquires biometric information of a user performing an operation by position tracking. The fatigue state determination unit (13) determines whether the user is in a fatigued state, on the basis of the biometric information acquired by the biometric information acquisition unit (12). The operation settings changing unit (14) fixes predetermined position tracking of the user when the fatigue state determination unit (13) determines that the user is in a fatigued state.

Description

Operating device, operating system, and operating method

[0001] The present disclosure relates to an operating device, an operating system, and an operating method.

[0002] In virtual reality (VR) technology, in order to move an avatar in a virtual space, a user needs to operate a controller for performing a movement operation or move their own hands and feet. In the method of moving an avatar by moving the user's hands and feet, changes in the user's position can be detected by an acceleration sensor, a gyro sensor, a motion sensor, etc., and the detected changes in position can be reflected in the movement of the avatar. Such reflection of changes in the user's position on the avatar is called position tracking in order to track the user's position.

[0003] Regarding VR technology, Patent Document 1 discloses an information processing device aimed at easily realizing an effective display that can reduce symptoms such as 3D dizziness and fatigue that may occur in an experiencer. The information processing device described in Patent Document 1 includes an index acquisition means for acquiring an index representing symptoms that may occur in an experiencer experiencing a virtual space, a setting acquisition means for acquiring a display setting of an image viewed by the experiencer, and a notification means for notifying the experiencer of the acquired index and display setting.

[0004] Japanese Unexamined Patent Application Publication No. 2021-92883

[0005] However, a position tracking operation in a state where the same posture continues for a long time, such as an avatar operation in which the standing state continues for a long time, will give the user a sense of fatigue.

[0006] Therefore, development of a technology that can appropriately change the method of the position tracking operation according to the user's sense of fatigue is desired. Note that the technology described in Patent Document 1 cannot solve such a problem.

[0007] The operating device according to this disclosure includes a biometric information acquisition unit that acquires biometric information of a user performing an operation by position tracking, a fatigue state determination unit that determines whether or not the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit, and an operation setting change unit that fixes a predetermined position tracking of the user if the fatigue state determination unit determines that the user is in a fatigued state.

[0008] The operating system according to this disclosure comprises an operating target device and an operating device for operating the operating target device, wherein the operating device includes a biometric information acquisition unit for acquiring biometric information of a user performing position tracking operations on the operating target device, a fatigue state determination unit for determining whether the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit, and an operation setting change unit for fixing a predetermined position tracking of the user if the fatigue state determination unit determines that the user is in a fatigued state.

[0009] The operation method described herein involves the operation device acquiring biometric information of a user performing an operation using position tracking, determining whether the user is fatigued based on the acquired biometric information, and, if it is determined that the user is fatigued, fixing the user's predetermined position tracking.

[0010] According to this disclosure, the method of position tracking operation can be appropriately changed according to the user's level of fatigue.

[0011] This is a block diagram showing an example configuration of the operating system according to Embodiment 1. This is a schematic diagram showing an example of the operating system in Figure 1. This is a schematic diagram showing an example of the relationship between a user and an avatar using the operating system in Figure 1. This is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system in Figure 1. This is a flowchart illustrating an example of the position tracking process performed by the operating system in Figure 1. This is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system in Figure 1. This is a flowchart illustrating another example of the position tracking process performed by the operating system in Figure 1.

[0012] The following describes embodiments of the invention, but the invention claimed is not limited to these embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate.

[0013] <Embodiment 1> An example configuration of the operating system according to Embodiment 1 will be described using Figure 1. Figure 1 is a block diagram showing an example configuration of the operating system according to Embodiment 1.

[0014] As shown in Figure 1, the operating system according to this embodiment comprises an operating device 1 and an operating target device 3. The operating device 1 is a device that operates the operating target device 3, and this operation includes position tracking operation, which is an operation by position tracking.

[0015] The operating device 1 and the controlled device 3 are directly connected via wired or wireless communication, enabling them to send and receive information related to the operation. The operating device 1 and the controlled device 3 may communicate via a wired or wireless network such as a LAN, or they may communicate directly via cable or short-range wireless communication.

[0016] This explanation assumes that the operating device 1 and the controlled device 3 are separate devices, but they can be configured as a single device. Furthermore, for example, the controlled device 3 may have only some of the functions of the operating device 1, with the remaining functions connected to a separate device. The following describes examples of configurations for the controlled device 3 and the operating device 1.

[0017] (Operation target device 3) The operation target device 3 may include a control unit 30, a storage unit 36, a communication unit 37, and a display unit 38.

[0018] The target device 3 can be any device capable of position tracking operation, regardless of its type. The target device 3 can be, for example, a device that provides live music in a virtual space, a game device that provides games in a virtual space, or a communication device that allows communication with other users in a virtual space. Of course, even if the target device 3 is a game device, it can also have the functionality of a communication device. The target device 3 makes it possible to move an avatar that resembles the user in the virtual space.

[0019] The device to be operated 3 can include portable or stationary computers such as HMDs (Head Mounted Displays), mobile phones, smartphones, mobile PCs (Personal Computers), and PCs. As illustrated with the computer example, the device to be operated 3 may be an information processing device.

[0020] The device to be operated 3 can also be constructed as a system in which functions are distributed among multiple devices. For example, if the device to be operated 3 is configured to include an HMD equipped with a display unit 38, the device to be operated 3 may include a portable or stationary computer connected to the HMD by wire or wireless connection and on which application programs such as games are installed. Alternatively, for example, the device to be operated 3 may include a computer installed on the user's side and a server device connected to that computer via a network that provides the computer with application programs such as games that provide a virtual space to the user.

[0021] The control unit 30 controls the entire target device 3. The control unit 30 may be implemented by, for example, a processor such as a CPU (Central Processing Unit), working memory, and a non-volatile storage device that stores a control program. This program may include a program that, when executed by the processor, realizes the functions of the target device 3. This program may include application programs such as games. The control unit 30 may also be implemented using a programmable integrated circuit, such as an FPGA (field-programmable gate array) or a microcomputer.

[0022] The control unit 30 may include a main control unit 31, an input control unit 32, and a display control unit 33, which are functional blocks realized in the target device 3 by the operation of a program, etc., and are described later. The main control unit 31 may include an application program such as a game, and realizes various functions of the target device 3 while controlling the input control unit 32 and the display control unit 33.

[0023] The memory unit 36 ​​is a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive) that stores various types of information used by the device to be operated 3. As one type of information to be stored, the memory unit 36 ​​stores, for example, user setting information in application programs such as games. The input control unit 32 reads information stored in the memory unit 36 ​​and updates information stored in the memory unit 36.

[0024] The communication unit 37 communicates with external devices such as the operating device 1. The communication unit 37 is, for example, a communication unit that communicates in accordance with wireless communication standards such as Wi-Fi (registered trademark; the same applies hereinafter), or a communication unit that communicates in accordance with short-range wireless communication standards such as Bluetooth (registered trademark; the same applies hereinafter). The communication unit 37 may also be equipped with a communication unit for any mobile communication system in order to communicate with external devices other than the operating device 1.

[0025] The input control unit 32 sends and receives information with external devices such as the operating device 1 via the communication unit 37. When the input control unit 32 receives information indicating the operation content from the operating device 1, it passes that operation content to the main control unit 31. The main control unit 31 controls the display control unit 33 and other components according to this operation content.

[0026] The display unit 38 may be composed of a display device such as a liquid crystal panel or an organic electroluminescent panel. The display control unit 33 controls the display on the display unit 38 and changes the image representing the virtual space displayed on the display unit 38. For example, the display control unit 33 changes the image to be displayed on the display unit 38 according to the control from the main control unit 31 that corresponds to the operation content indicated by the information received from the operating device 1.

[0027] (Operating device 1) The operating device 1 detects when the user is fatigued based on the user's biometric information and, in response to the fatigued state, changes the settings so that the user can continue to experience the virtual space as if they were healthy. For this purpose, the operating device 1 may include a control unit 10, a storage unit 16, a communication unit 17, an input unit 18, a motion sensor 19, and a biosensor 20. The operating device 1 can be any device capable of accepting position tracking operations, regardless of its type or configuration.

[0028] The control unit 10 controls the entire operating device 1. The control unit 10 may be implemented by, for example, a processor such as a CPU, working memory, and a non-volatile storage device that stores a control program. This program may include a program that, when executed by the processor, realizes the functions of the operating device 1. Alternatively, the control unit 10 may be implemented using an integrated circuit that can be programmed by the user, such as an FPGA or a microcomputer.

[0029] The control unit 10 may include a tracking processing unit 11, a biological information acquisition unit 12, a fatigue state determination unit 13, and an operation setting change unit 14, which are functional blocks realized by the operation of a program in the operating device 1. The control unit 10 may also include a posture change determination unit 15, and a configuration including the posture change determination unit 15 will be described later as Embodiment 2.

[0030] The storage unit 16 is a storage device such as an SSD or HDD that stores various types of information used in the operating device 1. As one type of information to be stored, the storage unit 16 stores, for example, user setting information in the operating device 1.

[0031] The communication unit 17 communicates with the device to be operated 3. The communication unit 17 may be a communication unit such as the one exemplified as the communication unit 37.

[0032] The input unit 18 is a part that accepts user input and can also be called the operation unit. The input unit 18 is preferably provided to supplement the movement operation of the avatar by the motion sensor 19, but the operation device 1 may be configured without an input unit 18. The input unit 18 may consist of at least one of a physical key and a software key, or it may consist of a separately provided voice input unit and voice recognition unit, or it may consist of both. The input unit 18 may also include a touch sensor incorporated into a separately provided display unit (not shown). In that case, the operation device 1 will include a display unit and a touch panel as an example of the input unit 18.

[0033] The motion sensor 19 is one or more sensors that detect the user's movements, and can be composed of, for example, a gyroscope and an accelerometer. The motion sensor 19 can be, for example, a 6DoF (Degree of Freedom) sensor, but it is sufficient that it can detect changes in the position of one or more parts of the user's body, that is, the movement of one or more parts of the user. The motion sensor 19 may also include a motion capture camera installed at a location separate from the main body of the operating device 1.

[0034] Furthermore, if the device to be operated 3 includes an HMD equipped with a display unit 38, at least a portion of the motion sensor 19 of the operating device 1 may be mounted on the HMD as a sensor for detecting head movement. In this case, the motion sensor 19 may include not only position tracking sensors, but also head tracking sensors, eye tracking sensors, etc. Note that some or all of the head tracking sensors may be position tracking sensors.

[0035] The motion sensor 19 transmits information indicating the detected user movement to the tracking processing unit 11. Based on the user movement detected by the motion sensor 19, the tracking processing unit 11 determines the content of the position tracking operation and outputs information indicating the content of that operation to the target device 3 via the communication unit 17. The tracking processing unit 11, or the tracking processing unit 11 and the communication unit 17, can be referred to as a VR controller, for example, because they instruct the user's movements to be reflected in the virtual space.

[0036] The input unit 18 can consist of, for example, a key for turning the power on and off, but it can also include a key that accepts user movement and direction of movement instructions instead of detecting user movement with the motion sensor 19. In that case, the control unit 10 should transmit these instructions to the device to be operated 3 via the communication unit 17.

[0037] The biosensor 20 is a group of sensors consisting of one or more types of sensors that acquire the user's biometric information, or a device equipped with such a group of sensors. For example, the biosensor 20 can be mounted on wearable devices such as bracelets, rings, glasses, or in-ear devices. Examples of such wearable devices include smartwatches, smart rings, smart glasses, and smart earphones. Furthermore, if the device to be operated 3 includes an HMD equipped with a display unit 38, at least a portion of the biosensor 20 of the operating device 1 can be mounted on the HMD.

[0038] Furthermore, if at least a portion of the motion sensor 19 or biosensor 20 is mounted in a separate housing from the main body of the operating device 1, it is preferable that the separate housing be equipped with a communication unit that communicates wirelessly or via wired connection with the main body in order to transmit the detection results to the main body.

[0039] The biosensor 20 detects biological information that allows for the estimation of the user's fatigue level or whether the user is fatigued, based on its detection result or the temporal change in its detection result. Biological information is also called vital information. Biological information may be one or more of various pieces of information, such as pulse waves showing fluctuations in pulse rate, body temperature, blood pressure, respiratory rate, electroencephalogram, and cerebral blood flow. Cerebral blood flow can be detected, for example, by an optical sensor mounted on an in-ear wearable device worn on the ear, or by an optical sensor mounted near the ear of an HMD. The biosensor 20 may also include, for example, an audio input unit consisting of a microphone for inputting the user's voice, or a camera for capturing images of the user's face.

[0040] The biometric information acquisition unit 12 acquires biometric information from the biosensor 20 and passes it to the fatigue state determination unit 13. In particular, the biometric information acquisition unit 12 acquires biometric information of the user performing position tracking operations on the target device 3 and passes it to the fatigue state determination unit 13. The biometric information acquisition unit 12 may generate biometric information in a format to be used for determination by the fatigue state determination unit 13 by, for example, extracting characteristic quantities of the biometric information from the acquired biometric information, and then pass it to the fatigue state determination unit 13.

[0041] Feature quantities of biometric information can be values ​​that represent the characteristics of the waveform of the pulse wave, for example, if the acquired biometric information is pulse wave information, and similarly for other types of biometric information, they can be values ​​that represent the characteristics of that biometric information. If the biosensor 20 is equipped with an audio input unit, the biometric information acquisition unit 12 may extract feature quantities or changes in feature quantities such as the intonation and volume of the user's voice from the audio data acquired by the audio input unit. If the biosensor 20 is equipped with a camera, the biometric information acquisition unit 12 may extract feature quantities or changes in feature quantities such as the user's facial expression, gaze, and pupil size from the image data captured by the camera. Feature quantities of facial expression can refer to features such as the angle of the corners of the mouth and the corners of the eyes.

[0042] Furthermore, in order to extract features from biological information, the biological information acquisition unit 12 may be equipped with a pre-trained model that has been trained to take the biological information received from the biological sensor 20 as input and output features. The algorithm and other aspects of this pre-trained model are not limited.

[0043] The fatigue state determination unit 13 determines whether the user is fatigued or not based on the biometric information received from the biometric information acquisition unit 12. For example, the fatigue state determination unit 13 calculates the user's fatigue level based on the biometric information received from the biometric information acquisition unit 12, and determines whether the user is fatigued or not based on whether that fatigue level is above a threshold. The fatigue state determination unit 13 passes the determination result to the operation setting change unit 14.

[0044] For example, when the brain wave information indicates that the sympathetic nerve is dominant, if the brain wave is a beta wave, the fatigue state determination unit 13 may determine that the mental state is an excited state, while if it is a gamma wave, it may determine that the mental state is a concentrated state. Then, if the duration of the excited state is longer than the first predetermined period, the fatigue state determination unit 13 determines that the mental state is a stress state, and if the stress state continues for a second predetermined period longer than the first predetermined period, it is estimated that the user is in a fatigue state. Although only brain wave information has been exemplified, the fatigue state determination unit <SP>13< / SP> can perform similar determination from biological information other than brain wave information based on the relationship between the biological information and the fatigue state. For example, when the cerebral blood flow is lower than a predetermined amount, the fatigue state determination unit <SP>13< / SP> can determine that the user is in a fatigue state.

[0045] In order to calculate the degree of fatigue or determine whether the user is in a fatigue state, the fatigue state determination unit <SP>13< / SP> may be provided with a trained model that is machine-learned to input biological information and output the degree of fatigue or whether the user is in a fatigue state. The algorithm or the like of this trained model is not limited.

[0046] The operation setting change unit <SP>...< / SP> receives the determination result from the fatigue state determination unit <SP>13< / SP>. Then, when it is determined that the user is in a fatigue state, the operation setting change unit <SP>14< / SP> fixes the predetermined position tracking of the user. That is, when it is determined that the user is in a fatigue state, the operation setting change unit <SP>1< / SP><SP>4< / SP> changes the operation setting so as to fix the predetermined position tracking of the user. The information indicating the operation setting can be stored in the storage unit <SP>16< / SP> as a part of the above-described user setting information.

[0047] The predetermined position tracking can be, for example, position tracking in the height direction, that is, position tracking in the vertical direction, but is not limited to this, and may be position tracking in the left-right direction or the like.

[0048] Fixing the vertical position tracking means not reflecting the change in the height direction in the user's position tracking. Not reflecting the change in the vertical direction is beneficial, for example, in a virtual space where an event such as a music live is held and the user participates while standing and swaying the body. Fixing the horizontal position tracking means not reflecting the change in the horizontal direction in the user's position tracking. Not reflecting the change in the horizontal direction is beneficial, for example, in a virtual space where an event such as a graduation ceremony is held and the user's left - right swaying is prominent.

[0049] Also, for a predetermined position tracking, the part to be fixed may be predetermined in advance, such as an upper - body part in the case of the vertical direction. Also, for the part to be fixed, for example, even when fixing the vertical position tracking, only the central position of that part may be fixed. Thereby, for example, even when fixing the vertical position tracking of the head, the user's nodding motion can be tracked and reflected in the avatar.

[0050] Also, the predetermined position tracking can be varied according to the type of space represented in the virtual space, such as a live venue, a bedroom, a plateau, etc., that is, according to the situation of the provided virtual space. Also, the process of fixing the predetermined position tracking may be set to be executable or not for each user, or the threshold value for fatigue determination may be set for each user. Also, the content of such settings may be stored in the storage unit 16 as user - specific user - setting information.

[0051] An example of the application of such an operating system and its effects will be briefly explained. As an example, consider a case where a user is participating in a VR live event. In this case, if the user stands for a long time participating in the VR live event, fatigue will accumulate in the user's legs. At that time, the biometric information acquisition unit 12 acquires biometric information from one or more biosensors 20, such as a microphone for acquiring sound, a wearable device for measuring brain waves, and a camera for capturing facial expressions, for the user participating in the VR live event. The fatigue state determination unit 13 then determines from this biometric information whether or not the user is tired. The fatigue state determination unit 13 will determine that the user is in a fatigued state if the user has been standing for a long time to the point where fatigue has accumulated in their legs.

[0052] In this way, if the system determines that the user is fatigued, the operation setting change unit 14 changes the operation settings to disconnect the position tracking of the user's feet and body height, thereby fixing the position tracking of the user's feet and body height. Disconnecting position tracking here means that the tracking processing unit 11 does not reflect the content of the position tracking operation determined based on the information indicating the user's movement detected by the motion sensor 19 onto the avatar in the virtual space. In other words, when the operation setting is changed as described above, the tracking processing unit 11, or the tracking processing unit 11 and the communication unit 17, do not reflect the change in foot and body height detected by the motion sensor 19 onto the avatar in the virtual space. Fixing position tracking means not changing the coordinates of the position at the time the user is determined to be fatigued, or at the time position tracking is disconnected. For this reason, the tracking processing unit 11 may, for example, store the coordinates of the foot and body height at the time the user is determined to be fatigued, or at the time position tracking is disconnected, in the storage unit 16, and continue to use those coordinates without updating them thereafter.

[0053] Specifically, if the fatigue level measurement indicates that the user is fatigued, the operation setting change unit 14 changes the method of operating the avatar in the VR live space so that the height of the feet and body, controlled by the gyro sensor or HMD position tracking, is not reflected in the avatar in the virtual space. In such an operation system, for example, by disconnecting the position tracking of a fatigued user, it is possible to suppress the shaking of the virtual space image or avatar displayed on the display unit 38 in response to the user being unsteady or swaying.

[0054] Next, using Figures 2 to 4, we will describe an example of an operating system in which the main part of the operating device 1 and the operating target device 3 are integrated. Of course, the operating system shown in Figure 2 is merely one example, and other configurations can also be applied. Figure 2 is a schematic diagram showing an example of the operating system in Figure 1. Figure 3 is a schematic diagram showing an example of the relationship between a user and an avatar using the operating system in Figure 1. Figure 4 is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system in Figure 1.

[0055] The operating system shown in Figure 2 comprises an HMD2 worn on the head of user U, a sensor 19b which is part of a motion sensor 19 installed at a distance from user U, and a sensor 20b which is part of a biosensor 20 worn on user U's arm. Sensor 19b is a motion capture sensor that detects user U's movements as changes in the position of various parts of user U, such as those shown by the black circles in the diagram. In Figure 2, an example is shown where sensor 19b is installed on a table, but the installation method is not limited to hanging from the ceiling, etc. Sensor 20b is a wearable device that measures user U's pulse, for example.

[0056] The HMD2 is equipped with the device to be operated 3 and the main part of the operating device 1. The main part of the operating device 1 mounted on the HMD2 includes a sensor 19a, which is part of the motion sensor 19, and a sensor 20a, which is part of the biosensor 20. Sensor 19a is, for example, a gyro sensor and an acceleration sensor. Sensor 20a is, for example, an optical sensor that detects cerebral blood flow.

[0057] In Figure 3, User Ua represents User U in a standing position, and for convenience, Avatar AVa is shown alongside User Ua. Avatar AVa is a virtual avatar in space resulting from tracking User Ua with the motion sensor 19. User Ua and Avatar AVa coincide in the height direction at the body part targeted by position tracking, indicated by the black circles.

[0058] Figure 4 shows a scenario where user U has accumulated fatigue and is sitting in chair CH. In Figure 4, user U in the seated position is represented as user Ub, and for convenience, avatar AVb is shown next to user Ub.

[0059] Avatar AVb is an avatar in the virtual space resulting from tracking the user Ub with the motion sensor 19, and is the avatar when settings are changed in the operation setting change unit 14 in this embodiment. In Figure 4, it can be seen that the user Ub and avatar AVb are misaligned in the height direction in at least a part of the body targeted for position tracking, indicated by the black circles, specifically in the upper body including the head, hands, and torso.

[0060] This is because a setting change was made that fixed the vertical position tracking of user Ub. More specifically, the operation setting change unit 14 fixed avatar AVb in the same standing position as avatar AVa corresponding to user Ua in a standing position, and did not reflect the change in upper body height when user Ua became user Ub. In this case, position tracking other than vertical position tracking continues, and head tracking and eye tracking, if performed, also continue.

[0061] On the other hand, if this embodiment is not applied and no settings are changed in the operation setting change unit 14, an avatar in a seated position with the same posture as user Ub will be displayed (not shown in the figure).

[0062] (Processing in the Operating System) Next, an example of position tracking processing performed by the operating device 1 of the operating system in Figure 1 will be explained using Figure 5. Here again, the explanation will be given using an operating system including the HMD2 as shown in Figure 2. Figure 5 is a flowchart illustrating an example of such position tracking processing.

[0063] First, as shown in Figure 2, the user U puts on the HMD2 and sensor 20b and stands within the detection range of sensor 19b, so that the device to be operated 3 can be operated by the operating device 1. Next, the user U, for example, when standing, inputs a calibration execution operation from the input unit 18. The control unit 10 of the operating device 1 receives this calibration execution operation and performs the calibration (step S11).

[0064] Calibration involves initial setup, where the user U assumes the same pose as the displayed avatar, thereby synchronizing the user U's movements with those of the avatar. Performing initial setup in multiple postures is advantageous in improving the accuracy of reflecting the user U's movements to the avatar. Therefore, calibration should be performed in various scenarios where the user U assumes multiple postures, such as standing and sitting.

[0065] In step S11, specifically, the tracking processing unit 11 instructs the motion sensor 19, which consists of sensors 19a and 19b, to perform detection, and registers the initial values ​​such as the user's initial position and initial angle detected by the motion sensor 19 in the storage unit 16 as part of the user setting information. If eye tracking is performed, the tracking processing unit 11 may also register the initial value of the user's gaze detected by the eye tracking sensor in the storage unit 16 as part of the user setting information. In addition, the tracking processing unit 11 may perform room scale measurement to measure the floor position, room size, etc., and register these in the storage unit 16 as part of the user setting information.

[0066] Furthermore, calibration may include, if necessary, superimposing an avatar corresponding to the user U onto an image representing the virtual space on the display unit 38 of the device to be operated 3, and registering the avatar's position and size as part of the user setting information in the storage unit 16.

[0067] Furthermore, calibration may include calibration of the biosensor 20. Specifically, when a calibration operation is received, the bioinformation acquisition unit 12 instructs the biosensor 20 to detect bioinformation, and obtains initial values ​​of the bioinformation by extracting its features from the biosensor 20. The bioinformation acquisition unit 12 then registers these initial values ​​in the storage unit 16 as part of the user setting information.

[0068] Once the calibration in step S11 is complete, the tracking processing unit 11 starts accepting tracking operations, including position tracking operations, and begins the tracking process (step S12). As a result, the display unit 38 of the target device 3 displays an avatar on the image representing the virtual space, reflecting the user's tracking operations relative to the user U's initial values.

[0069] As step S12 begins, the biometric information acquisition unit 12 acquires biometric information of user U performing the position tracking operation from the biosensor 20 (step S13). This acquisition only needs to be performed while the tracking operation continues, but it does not need to be performed after step S15, which will be described later.

[0070] Next, the fatigue state determination unit 13 calculates the fatigue level of user U based on the biological information acquired by the biological information acquisition unit 12, and determines whether user U is in a fatigued state based on whether the fatigue level is above a predetermined level (step S14). If the result in step S14 is NO, the process returns to step S13, and the acceptance of tracking operations and acquisition of biological information continues.

[0071] On the other hand, if the answer in step S14 is YES, that is, if it is determined that user U is in a fatigued state, the operation setting change unit 14 fixes predetermined position tracking, such as vertical position tracking of user U (step S15).

[0072] As a result, as illustrated in Figure 4, for example, even if user U is sitting in a chair, if they are fatigued, the vertical tracking of user U will be fixed, and the avatar will be generated and displayed based on other tracking.

[0073] Even after the processing in step S15, tracking, including position tracking, continues (step S16). However, with regard to position tracking, position tracking other than the predetermined position tracking will continue.

[0074] Next, the control unit 10 determines whether or not it has received a termination operation from the input unit 18 (step S17), and terminates the process if it has received one. If the result in step S17 is NO, it is advisable to continue tracking, including the position tracking in step S16.

[0075] Furthermore, although not shown in Figure 5, the acquisition of biological information in step S13 may be continued after step S15, and the operation setting change unit 14 may release the predetermined position tracking lock when the fatigue state determination unit 13 determines that user U is no longer fatigued.

[0076] As described above, according to this embodiment, the method of position tracking operation can be appropriately changed depending on whether the user is energetic or tired, that is, according to the user's level of fatigue. As a result, according to this embodiment, for example, it is possible to suppress the shaking of the virtual space image or avatar displayed on the display unit 38 in response to the user being unsteady due to fatigue.

[0077] <Embodiment 2> The operating system according to Embodiment 2 will be explained using Figures 6 and 7, focusing on the differences from Embodiment 1, but various examples described in Embodiment 1 can be applied. Figure 6 is a schematic diagram showing another example of the relationship between a user and an avatar using the operating system of Figure 1.

[0078] In this embodiment, the control unit 10 includes a posture change determination unit 15. The posture change determination unit 15 determines whether or not the user's posture has changed based on the biological information acquired by the biological information acquisition unit 12. Examples of biological information used for this determination include blood pressure, heart rate, respiratory rate, and cerebral blood flow. Systolic and diastolic blood pressure can be used for this determination because they are slightly lower in a sitting position than in a standing position. Heart rate can be used for this determination because it may temporarily decrease when changing from a standing to a sitting position. Respiration is more stable in a sitting position than in a standing position, and the respiratory rate is lower in a sitting position than in a standing position, so these can be used for this determination. Cerebral blood flow can be used for this determination because it may temporarily increase when changing from a standing to a sitting position.

[0079] This change in posture can be defined by a predetermined state change. A predetermined state change may refer to, for example, a change from standing to sitting, a change from standing with both feet together to standing with one foot apart, or a change from standing to lying on one's back, on one's stomach, or on one's side. Other predetermined state changes may refer to, for example, changes between lying on one's stomach, on one's back, or on one's side, or a change from sitting to lying on one's back, on one's stomach, or on one's side. Each state can be predetermined based on the positional relationships between the coordinates of the user's various body parts. Even for changes other than between standing and sitting, biological information such as blood pressure, heart rate, respiratory rate, and cerebral blood flow may change, and if the trends of these changes are learned in advance, the state change can be determined from the changes in that biological information.

[0080] Furthermore, the posture change determination unit 15 may calculate and acquire the posture of the user wearing the HMD2, including the head position, head orientation, and movement, based on detection information from acceleration sensors, angular velocity sensors, etc., mounted on the HMD2. One posture calculation method is a posture estimation method that uses an inertial measurement unit (IMU) to estimate the posture. In this method, a gyroscope, accelerometer, and magnetometer are combined to acquire the three-dimensional coordinates of each part of the user and estimate the posture. Another posture calculation method is to use a depth sensor such as Kinect® to acquire the user's joint positions in real time and estimate the posture. Alternatively, the user's posture may be determined by performing image analysis using an external camera.

[0081] Thus, the posture change determination unit 15 can also calculate posture based on detection information from sensors other than the biosensor 20 and determine changes in posture based on the calculated results. Furthermore, the posture change determination unit 15 may determine changes in posture based on detection information from both the biosensor 20 and sensors other than the biosensor 20. For example, a trained model may be created that combines biometric information such as fluctuations in heart rate and respiratory rate, or changes in body temperature, with detection information from other sensors, and is machine-learned to take biometric information and detection information as input and output changes in posture. The posture change determination unit 15 may then obtain changes in posture based on this trained model. The algorithm of this trained model is not limited.

[0082] In this embodiment, the operation setting change unit 14 fixes the predetermined position tracking when the fatigue state determination unit 13 determines that the user is in a fatigued state and the posture change determination unit 15 determines that the user's posture has changed.

[0083] In the following, as with the example in Embodiment 1, we will provide an example where the predetermined position tracking is vertical position tracking, and an example where the change in posture is a change from a standing position to a sitting position.

[0084] If the fatigue state determination unit 13 determines that the user is fatigued and the posture change determination unit 15 determines that the user's posture has changed from standing to sitting, the operation setting change unit 14 performs the following fixation. Specifically, in this case, the operation setting change unit 14 fixes the vertical position tracking of the user's feet and body by changing the operation settings to disconnect the vertical position tracking of the user's feet and body. The coordinates used to fix the position tracking are the coordinates at the time the user was determined to be in a standing state.

[0085] This fixed positioning disconnects vertical position tracking, so vertical movements of the user's body are not reflected in the avatar. As a result, even if the user becomes tired and sits down, the avatar can maintain an upright position.

[0086] In this case, if position tracking is completely disabled, no body movements will be reflected in the avatar. Therefore, the information that the character is sitting is ignored, and other movements such as arm swings or body sway are reflected instead. In other words, in this embodiment, in the tracking control including position tracking, other movements are reflected without reflecting that the character is sitting.

[0087] More specifically, in this embodiment, when the user sits down due to fatigue, the height changes of each tracking point are not reflected, so the avatar's upper body maintains the same height as when standing. As a result, in this embodiment, the user's viewpoint height is maintained even when sitting, thus maintaining the field of view as when standing. Therefore, in this embodiment, the avatar can be operated with a standing gaze, allowing the user to continue enjoying VR live performances and other content.

[0088] To elaborate, if position tracking were functioning normally when a user changes from a standing to a sitting position, the user's eye level would also decrease. Therefore, in this embodiment, the eye level is maintained without change by disabling vertical position tracking. This maintenance is particularly beneficial in situations such as VR live performances, where it is necessary to maintain the user's gaze from a standing position even when they are sitting.

[0089] For example, it is assumed that users participating in a VR live event are standing for a long period of time, such as 30 minutes to an hour or more. In this case, if a user who has been standing for a long time changes their posture to sitting or lying down, it is thought that the user wanted to maintain the field of view from when they were standing, but changed their posture due to fatigue. Therefore, in this embodiment, the state of standing for a long time is defined as the first posture, and when it is determined that the posture has changed from the first posture, it is determined whether or not the user is tired, and if the user is tired, the height of the viewpoint is fixed. This prevents malfunctions such as fixing the height of the viewpoint when it is desired that the user's sitting or lying down movements be reflected in the avatar.

[0090] Furthermore, if the fatigue state determination unit 13 determines that the user is fatigued, the operation setting change unit 14 may reset the user's post-change posture as the initial value of the user's pre-change posture. The user's post-change posture refers to the posture in which the user has settled after changing their posture, that is, the posture that has been maintained for a predetermined period of time after the change. Also, resetting the user's posture as the initial value means resetting the initial value of the avatar's posture corresponding to the user, that is, performing position tracking calibration again. As a result, the difference between the value detected by the motion sensor 19 and the redefined initial value will be reflected in the avatar.

[0091] During the initial calibration, the user's movements are synchronized with those of the avatar by having the user assume the same pose. This means that even if the initial calibration is performed in various postures, the user's standing position is registered as the initial value for the avatar's upright position. Therefore, after the initial calibration, the user's standing position is usually linked to the avatar's standing position. Consequently, if a user changes to a seated position and tries to manipulate an avatar in a standing position using motion tracking, the resulting movement may be unintended by the user.

[0092] In contrast, as in this embodiment, if the user's posture after a change, such as sitting, is reset as the initial value of the user's posture before the change, the user's seated state and the avatar's standing state can be linked. In other words, such a reset means recalibrating the system by treating the user's seated state as if the user were standing. As a result, even if the user changes from a standing state to a seated state, if the user is fatigued, they can still operate the avatar in a standing state.

[0093] Let's explain this using Figure 6. In Figure 6, as in Figure 4, fatigue has accumulated in User U, and User U is sitting in chair CH. In this seated position, User U has extended their legs and put their feet forward. In Figure 6, User U in this seated position with their legs extended is shown as User Uc, and for convenience, Avatar AVc is shown next to User Uc.

[0094] Avatar AVc is an avatar in the virtual space resulting from tracking the user Uc with the motion sensor 19, and is the avatar when settings are changed in the operation setting change unit 14 in this embodiment. In Figure 6, as in Figure 4, it can be seen that the user Ub and avatar AVb are misaligned in the height direction in the head, hands, and torso, which are the targets of position tracking indicated by the black circles.

[0095] However, in Figure 6, when user U steps forward with their leg, as in user Uc, the avatar AVc performs a kicking motion with the hips as its axis. To enable the reflection of such movements, during recalibration, it is recommended to set up the avatar to associate, for example, the action of extending the lower leg while seated with the action of stepping forward with the leg. When this action is reflected in the avatar, the action of user U stepping forward with their leg, as in user Uc, will be reflected in the avatar AVc as a kicking motion with the hips as its axis.

[0096] Furthermore, through recalibration, the user Uc and the avatar AVc may be made the same in the height direction of the feet, as shown in Figure 6, or they may be made different according to the ratio between the height of the chair CH and the length of the user U's legs. If they are made different according to the ratio, it means that the avatar AVc can only raise the feet up to the height of the chair CH.

[0097] Figure 7 is a flowchart illustrating another example of the position tracking process performed in the operating system shown in Figure 1.

[0098] First, the same process as in steps S11 to S14 in Figure 5 is performed (steps S31 to S34). If the answer in step S34 is NO, the process returns to step S33, and the acceptance of tracking operations and acquisition of biological information continues.

[0099] On the other hand, if the answer in step S34 is YES, that is, if it is determined that user U is in a fatigued state, the posture change determination unit 15 determines whether or not the user's posture has changed (step S35). As described above, the determination in step S35 can be, for example, whether or not user U has changed from a standing position to a sitting position. If the answer in step S35 is NO, the process returns to step S33, and the acceptance of tracking operations and acquisition of biological information continues.

[0100] On the other hand, if the answer in step S35 is YES, that is, if it is determined that user U has changed from a standing position to a sitting position, the operation setting change unit 14 fixes predetermined position tracking, such as vertical position tracking of user U (step S36). The process in step S36 is the same as the process in step S15 in Figure 5.

[0101] Next, the control unit 10 of the operating device 1 performs calibration so that the avatar of the pre-change posture can be represented in the changed posture (step S37). The calibration in step S37 is a recalibration, but the calibration method itself is as described in step S11 of Figure 5. However, the recalibration differs from the initial calibration in that it is not triggered by a calibration operation from the user U, and it is performed so that the avatar of the pre-change posture can be represented in the changed posture. The calibration in step S37 may or may not include calibration of the biosensor 20.

[0102] Even after the processing in step S37, tracking, including position tracking, continues in the same way as in step S16 in Figure 5 (step S38). However, in step S38, tracking is performed based on the user setting information after calibration is performed again in step S37. Of course, in step S38 as well, position tracking other than the predetermined position tracking will continue.

[0103] Next, similar to step S17 in Figure 5, the control unit 10 determines whether or not it has received a termination operation from the input unit 18 (step S39), and terminates the process if it has received one. If the result in step S39 is NO, it is advisable to continue tracking, including the position tracking in step S38.

[0104] Furthermore, although not shown in Figure 7, the acquisition of biological information in step S33 may be continued after steps S36 and S37. Then, when user U is no longer fatigued or returns from a seated position to a standing position, the operation setting change unit 14 may release the fixed position tracking.

[0105] According to this embodiment, by determining changes in the user's posture and fixing a predetermined position tracking, it becomes unnecessary to fix the predetermined position tracking when there is no change in posture, such as when the user is tired but not to the point of changing their posture. Furthermore, according to this embodiment, by adopting a configuration that resets the initial values, the user can operate the avatar in the standing position even when sitting, and can operate the avatar in the state before the posture change even after changing their posture.

[0106] Note that while Figure 7 shows a recalibration process after a change in posture, this is not the only option. For example, the operation setting change unit 14 may associate the avatar's posture before the change with the position tracking results for the posture after the change and record them in a database format in the storage unit 16. This allows the tracking processing unit 11 to refer to this database and perform position tracking operations on the avatar as if the user's posture had not changed, even after the user has changed their posture. For example, the operation setting change unit 14 may associate and store the movements of a seated state with those of an upright state, allowing the user to operate the upright avatar even when seated.

[0107] Furthermore, a process that produces a similar effect to this example of resetting initial values ​​can also be applied to Embodiment 1, which does not involve determining changes in the user's posture or fixing a predetermined position tracking based on the determination result.

[0108] In other words, in Embodiment 1, if the fatigue state determination unit 13 determines that the user is fatigued, the operation setting change unit 14 may set the operation so that the user's avatar maintains the posture of the user before fatigue, while corresponding to the posture of the user after fatigue. The user's avatar is the target of operation by the user's position tracking. For example, the operation setting change unit 14 may associate the avatar's posture before the user became fatigued with the position tracking results of the user's posture after fatigue and record them in a database format in the storage unit 16. This allows the tracking processing unit 11 to refer to this database and perform position tracking operations on the avatar as if the user were not fatigued, even after the user has become fatigued.

[0109] <Embodiment 3> In Embodiments 1 and 2, the operation setting change unit 14 fixed the user's predetermined position tracking when it determined that the user was in a fatigued state. Such fixing is useful, for example, in VR live performances, but there are also virtual spaces where it is desirable to represent standing and sitting states with avatars.

[0110] Therefore, in this embodiment, the operation setting change unit 14 performs such fixing only when the determination result of the posture change determination unit 15 indicates that the user has remained in an upright position for a predetermined period of time, or when the same state has continued for a predetermined period of time. In other words, in this embodiment, the operation setting change unit 14 fixes a predetermined position tracking if the fatigue state determination unit 13 determines that the user is in a fatigued state and the posture change determination unit 15 determines that the user's posture has changed after remaining unchanged for a predetermined period of time.

[0111] According to this embodiment, except in cases where the user's posture has not changed for a long period of time, the avatar can represent any posture, such as standing or sitting, even when the user is tired.

[0112] Furthermore, in this embodiment, the predetermined position tracking is fixed when the user becomes fatigued, but in a further application example, such fixing may be performed without considering the user's fatigue. That is, in this application example, the fatigue state determination unit 13 and the biosensor 20 do not need to be provided in the operating device 1. In this application example, the operation setting change unit 14 fixes the predetermined position tracking when the posture change determination unit 15 determines that the user's posture has changed after remaining unchanged for a predetermined period of time.

[0113] As shown in these application examples, by determining whether or not to perform a process to fix a predetermined position tracking based solely on the time related to changes in posture, without considering the user's fatigue state, it is possible to fix the desired position tracking regardless of the user's fatigue state.

[0114] <Alternative Examples, etc.> Some or all of the processing in the operating device 1, HMD 2, or target device 3 described above can be implemented as a computer program. The program described above includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments when loaded into a computer. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrical, optical, acoustic or other forms of propagating signals.

[0115] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

[0116] For example, the biometric information mentioned above can be other information than that exemplified, and the position tracking method and the type of device being operated are not limited to those exemplified. Furthermore, the above embodiment was described on the premise that appropriate modifications to the position tracking operation method would be applied to VR technology. However, such appropriate modifications to the position tracking operation method may also be applied to mixed reality (MR) technology or augmented reality (AR) technology, which merge real space and virtual space in real time.

[0117] This application claims priority based on Japanese Patent Application No. 2024-158152, filed on 12 September 2024, and incorporates all of its disclosures herein.

[0118] This disclosure is applicable to various technologies for performing position tracking operations.

[0119] 1 Operating device 2 HMD 3 Device to be operated 10, 30 Control unit 11 Tracking processing unit 12 Biological information acquisition unit 13 Fatigue state determination unit 14 Operation setting change unit 15 Posture change determination unit 16, 36 Storage unit 17, 37 Communication unit 18 Input unit 19 Motion sensor 19a, 19b Sensor 20 Biological sensor 20a, 20b Sensor 31 Main control unit 32 Input control unit 33 Display control unit 38 Display unit U, Ua, Ub, Uc User AVa, AVb, AVc Avatar

Claims

1. An operating device comprising: a biometric information acquisition unit that acquires biometric information of a user performing an operation using position tracking; a fatigue state determination unit that determines whether or not the user is in a fatigued state based on the biometric information acquired by the biometric information acquisition unit; and an operation setting change unit that fixes a predetermined position tracking of the user if the fatigue state determination unit determines that the user is in a fatigued state.

2. The operating device according to claim 1, further comprising a posture change determination unit that determines whether or not the user's posture has changed based on the biological information acquired by the biological information acquisition unit, wherein the operation setting change unit fixes the predetermined position tracking when the fatigue state determination unit determines that the user is in a fatigued state and the posture change determination unit determines that the user's posture has changed.

3. The operation setting change unit, when the fatigue state determination unit determines that the user is in a fatigued state, sets the operation so that the avatar targeted by the user's position tracking maintains the user's posture before fatigue while corresponding to the user's posture after fatigue, as described in claim 1 or 2.

4. The operating device according to any one of claims 1 to 3, wherein the predetermined position tracking to be fixed varies depending on the type of space represented in the virtual space.

5. Fixing the position tracking means not changing the coordinates of the position at the time when the user is determined to be in a fatigued state or when the position tracking is disconnected, according to any one of claims 1 to 4.

6. The operation setting change unit, when the fatigue state determination unit determines that the user is in a fatigued state, resets the user's changed posture to the initial value of the user's posture before the change, as described in claim 3.

7. The operation setting change unit records the correspondence between the avatar in the state of posture before the change and the position tracking result of the state of posture after the change, and performs the operation setting by referring to the correspondence, as described in claim 3.

8. The operating device according to claim 3, further comprising a posture change determination unit that determines whether or not the user's posture has changed based on the biological information acquired by the biological information acquisition unit, wherein if the same state continues for a predetermined period of time as determined by the posture change determination unit, the operation setting change unit performs the predetermined position tracking fixation.

9. An operating system comprising: an operating target device; and an operating device for operating the operating target device, wherein the operating device includes: a biometric information acquisition unit for acquiring biometric information of a user performing position tracking operations on the operating target device; a fatigue state determination unit for determining whether the user is fatigued based on the biometric information acquired by the biometric information acquisition unit; and an operation setting change unit for fixing a predetermined position tracking for the user if the fatigue state determination unit determines that the user is fatigued.

10. An operating method in which an operating device performs the following processes: acquires biometric information of a user performing an operation using position tracking; determines whether the user is fatigued based on the acquired biometric information; and, if it is determined that the user is fatigued, fixes the user's predetermined position tracking.

Images