Head-mounted information processing device

Abstract

To provide a head-mounted information processing device that warns a user about mistakenly perceiving a virtual space as a real space.SOLUTION: A head-mounted information processing device includes a display unit, a surroundings monitoring sensor that detects an object present around a user, a user state determination unit that determines that the user's state is not suitable for viewing virtual space information when an approaching object is deemed to be present within a certain range of the user on the basis of the detection result of the surroundings monitoring sensor, and a control unit, and when the user state determination unit determines that the user's state is suitable for viewing virtual space information, the control unit causes the display unit to display the virtual space information in a normal display mode, and when the control unit determines that the state is not suitable, the control unit instructs the display unit to display the virtual space information in a specific display mode or to suspend display.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a head-mounted information processing device, and more particularly to a display mode of virtual space information. [Background technology]

[0002] The virtual space information (virtual objects) is displayed on the head-mounted display worn by the user. There is a technology that displays a virtual image (a virtual object) and superimposes it on the real world so that the user can see it. According to Und Display, the real and virtual worlds will be seamlessly integrated in real time. This allows the user to experience the virtual object as if it were actually there. Users can experience a deep sense of immersion, as if they are mistaken for the real world. However, on the other hand, deep immersion allows users to easily connect the virtual world with reality. The problem was that it became difficult to distinguish and differentiate between the world.

[0003] Regarding this issue, Patent Document 1 states that "information including at least a part of a virtual object Feeds to users based on the user's state as they experience the world provided The document states that the invention includes a feedback determination unit that determines the feedback. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 006640 Summary of the Invention [Problem to be solved by the invention]

[0005] For users wearing a head-mounted display, it is necessary to recognize the virtual space and the real space. The attention required to use a head-mounted display varies depending on the situation. For example, when stationary in the space in which we spend our daily lives, virtual objects, such as Even if the virtual cups and vases displayed on the table are mistaken for real objects, the user will not be affected. On the other hand, it is difficult to imagine that this is a significant issue for people walking outdoors. When spraying, even if there was an abandoned bicycle on the side of the sidewalk, the spray spread widely and no obstacles were found. If a virtual sidewalk is displayed, it may be mistaken for a real space, resulting in tripping over an abandoned bicycle. There are also.

[0006] In this way, the user is asked to avoid misunderstanding between the real space and the virtual space. The attention level is determined not only by the user's own attention level but also by the state of the surrounding environment. In this regard, Patent Document 1 describes a method for monitoring only the user's status and not the surroundings. The environment is not taken into consideration.

[0007] The present invention has been made in view of the above-mentioned circumstances, and aims to prevent a user from mistakenly recognizing a virtual space as a real space. The present invention aims to provide a head-mounted information processing device that alerts the user to the risk of injury. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention has the configurations described in the claims. head A mount information processing device, table a display unit; a camera that captures an image of the real space in front of the head-mounted information processing device, a gaze sensor that detects the gaze of the user of the head-mounted information processing device, and a control Equipped with the Imperial Department, The aforementioned The control unit a display unit that displays a virtual object and a real object included in the real space captured by the camera on the display unit; a display unit that displays a virtual object and a real object ... Display in normal display mode, When it is determined that the user is gazing at the real object, the virtual object is The display unit is instructed to display in a specific display mode or to suspend display. [Effects of the Invention]

[0009] According to the present invention, a user is warned about mistaking a virtual space for a real space. It is possible to provide a head-mounted information processing device that provides the user with the information. Objects, configurations, and effects other than those described above will become apparent from the following embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic view illustrating an external appearance of a head-mounted display system according to an embodiment of the present invention; [Figure 2] A block diagram showing an example of the configuration of the head-mounted display system shown in Figure 1. [Figure 3] Functional block diagram showing the functions of the main controller [Figure 4] Flowchart showing the processing flow of the head-mounted display system [Figure 5] FIG. 10 is a diagram showing types of display modes determined by a display control unit. [Figure 6] A diagram showing the coordinate system when viewing real space from a camera [Figure 7] A diagram showing the positional relationship between the object being gazed at and both eyes DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. The same symbols are used for the processes, and duplicate explanations will be omitted.

[0012] FIG. 1 shows an HMD system 1 (HMD: head-mounted display) according to this embodiment. 1, the HMD system 1 is an HMD worn on the head of a user. MD100, a first wearable terminal 200 (chest-mounted wear) worn on the chest of a user a second wearable terminal 300 (wristband type) worn on the user's arm; The first wearable terminal includes an input controller 400. The terminal 200, the second wearable terminal 300, and the input controller 400 are each a short-range wireless It is connected to the HMD 100 via communication and transmits and receives information. You may also make a statement.

[0013] The server 500 generates virtual space information and transmits it to the HMD 100 via a wireless communication line. The HMD 100 may be configured to generate and display virtual space information by itself.

[0014] The HMD 100 includes a camera 111, a right eye gaze sensor 112, a left eye gaze sensor 113, and an acceleration a temperature sensor 114, a gyro sensor 115, a geomagnetic sensor 116, a temperature and humidity sensor 117, and The surroundings monitoring sensor 118 is provided to monitor the user's front image (including the image of the surrounding objects), the user's line of sight, and It detects lines, the movement of the user's head, and the surrounding temperature and humidity.

[0015] Furthermore, the HMD 100 includes a display 119 placed in front of both eyes, and a camera 1 11 and the image of the real space information captured by the server 500. Show 30.

[0016] The HMD 100 also includes an external sound microphone 120 and a speech sound microphone 121. It collects voices and the user's own speech.

[0017] The HMD 100 is equipped with headphones 122 placed near the user's ears, and is used to receive audio and music from the headset. Output from phone 122.

[0018] Furthermore, the HMD 100 is equipped with an antenna 129. The server 500 transmits and receives information to and from the HMD 10, and the virtual space information generated by the server 500 is displayed on the HMD 10. 0 receives it and displays it on the display 119.

[0019] The camera 111 functions as a sensor that detects the user's surroundings and detects the user's line of sight. When used for visual inspection, it also functions as a sensor to detect the user's state. sensor 112, left eye gaze sensor 113, acceleration sensor 114, and gyro sensor 115. Each corresponds to a condition monitoring sensor.

[0020] The first wearable device 200 is equipped with a heartbeat sensor 201 and measures the number of heartbeats within a certain period of time. The heart rate, which is the number of times the heart beats, is detected.

[0021] The second wearable terminal 300 includes a blood pressure sensor 301 and detects the user's blood pressure. The second wearable device 300 is equipped with a pulse sensor, and measures the pulse rate, which is the number of times the artery beats. It may be detected.

[0022] The first wearable device 200 and the second wearable device 300 each have a status of the user. It corresponds to a monitoring sensor.

[0023] The input controller 400 is used by the user to perform various input operations. Based on the user's input operation received by the camera 400, the display provided in front of the user's eyes In the play 119, the image 130 of the virtual space information generated by the server 500 is added to the real space information. The HMD 100 itself generates a virtual space information image 130 and displays it in a superimposed manner, or displays real space information. It will be possible to switch from real-world information to virtual space information and display it.

[0024] The HMD 100 according to this embodiment is an HMD 100 worn closely by a user or a first Various sensors provided in the airable terminal 200 and the second wearable terminal 300 It detects the state of the user's body and the user's surroundings and adjusts the display based on both states. It is characterized in that the display mode of the image 130 of the virtual space information displayed on the screen 119 is changed. The display mode is a normal display mode when the user's physical condition and surrounding conditions are both normal. and a display mode when there is an abnormality in at least one of the user's physical condition and the surrounding condition. The present invention also provides a specific display mode that is different from the normal display mode. In the specification, one of the specific display modes includes non-display, and these will be described in detail later.

[0025] FIG. 2 is a block diagram showing an example of the configuration of the HMD system 1 shown in FIG.

[0026] In FIG. 2, the HMD 100 includes a camera 111, a right eye gaze sensor 112, a left eye gaze sensor 113, and a a temperature and humidity sensor 113, an acceleration sensor 114, a gyro sensor 115, a geomagnetic sensor 116, sensor 117, surrounding monitoring sensor 118, display 119, surrounding sound microphone 120, vocal sound A microphone 121, a headphone 122, a main body side vibrator 124, a main body side controller 125, A memory 128 for storing a program 126 and information data 127, an antenna 129, Body-side network communicator 131, body-side short-range wireless communication device 132 (corresponding to the body-side communicator) and a timer 133. Each of these components is configured using the antenna 129. They are connected to each other via a bus 140 .

[0027] The camera 111 is installed in front of the HMD 100 and captures the scenery in front of the user. The captured image is displayed on an immersive display 119 as an image in real space. When the ray 119 is a transmissive display, the user can see the real space information with their own eyes. Therefore, it is not necessary to display the image from the camera 111 on the display 119.

[0028] The right eye line of sight sensor 112 and the left eye line of sight sensor 113 detect the line of sight of the right eye and the left eye, respectively. The process of detecting the gaze is generally used as eye tracking processing. Well-known techniques can be used. For example, in a method using corneal reflection, an infrared LED (Light The face is illuminated with an infrared camera and the reflection caused by the infrared LED is measured. The position of the light on the cornea (corneal reflex) is used as the reference point, and the pupil position relative to the corneal reflex position is used as the reference point. A technology for detecting the line of sight is known.

[0029] The acceleration sensor 114 is a sensor that detects acceleration, which is a change in speed per second. It can capture movement, vibration, impact, etc.

[0030] The gyro sensor 115 is a sensor that detects the angular velocity in the rotation direction, and detects the vertical, horizontal, and diagonal The acceleration sensor installed in the HMD100 can capture the posture. 114 and a gyro sensor 115, the head of the user wearing the HMD 100 It can detect movement.

[0031] The geomagnetic sensor 116 is a sensor that detects the magnetic force of the earth, and is It detects the direction of the earth's magnetism in the vertical direction as well as the forward / backward and left / right directions. The head movement is detected by detecting the geomagnetic field change caused by the head movement. This allows for accurate estimation of head movement fluctuations while wearing the HMD100. can be detected in detail.

[0032] The temperature and humidity sensor 117 is a sensor that detects the temperature and humidity around the user.

[0033] The surroundings monitoring sensor 118 emits radio waves, light waves, ultrasonic waves, etc., and captures and reflects the reflected waves. It is a sensor that detects the distance and direction to an object in the user's sight. The distance and direction of the object around the user can be detected. , which shows various sensor devices.

[0034] The main body controller 125 constitutes a controller for the HMD 100 and stores data in the memory 128. The programs stored in the memory, such as the OS (Operating System) and the operation control applications, A processor that executes the RAM 126 and realizes the equivalent functions of these programs 126 The main body controller 125 controls each component of the HMD 100. It controls the overall operation of the HMD 100.

[0035] The main body controller 125 performs the following operations depending on whether or not there is an abnormality in the surrounding conditions and whether or not there is an abnormality in the user conditions: The virtual space information is superimposed on the real space information and displayed on the display 119, or The movement when converting spatial information into virtual spatial information and displaying it on the display 119 Control the operation.

[0036] The main controller 125 determines the user's Identify whether the physical state is abnormal, causing the user to mistakenly recognize virtual space information and real space information. The state of the user's periphery is used to determine the user's surroundings and to display virtual space information. It identifies and determines whether there is an abnormal condition that poses a risk.

[0037] Furthermore, the main body controller 125 may become unable to distinguish between virtual space information and real space information as the user's attention decreases. It is determined that this is an abnormal state in which the user mistakenly recognizes the virtual space information, and then superimposes the virtual space information on the real space information, or Display state when real space information is replaced with virtual space information and displayed on the display 119 The display mode is switched to the normal display mode or the specific display mode.

[0038] The main controller 125 detects objects around the user and detects if the objects are within a certain range of the user. When approaching the virtual space, an abnormal situation that is dangerous to the user occurs. When it is determined that there is an abnormality in the area, virtual space information is superimposed on real space information, or Display mode when real space information is replaced with virtual space information and displayed on the display 119 is switched from the normal specific display to a specific display mode or non-display.

[0039] The memory 128 is a flash memory or the like, and stores various data used by the main body controller 125. The program 126, the virtual space information transmitted from the server 500, and the first wearable The first sensor output received from the terminal 200 and the second sensor output received from the second wearable terminal 300 2. Information data 127 such as sensor output is stored.

[0040] The display 119 is configured with a liquid crystal panel or the like, and displays real space information and virtual space information. The image 130 is displayed, and the display contents such as notification information and operation status are displayed on the screen. For example, as shown in FIG. The captured image of the real space information and the image generated by the server 500 and transmitted to the HMD 100 The HMD 100 displays the received virtual space information on the screen in a superimposed manner on the image 130 .

[0041] The main body side vibrator 124 generates vibrations under the control of the main body side controller 125. This converts notification information sent by the HMD100 to vibrations for the user. The inflator 124 generates vibrations on the user's head when it is closely fitted to the user's head. This will ensure that notifications are delivered to the

[0042] The main body side network communication device 131 communicates via wireless LAN, wired LAN or base station communication. It is a communication interface for communicating with an external server 500, and The main body side network 600 is connected to the external network 600 via the antenna 129 to transmit and receive information. The network communication device 131 receives the data generated by the server 500 via the external network 600 or the like. The virtual space information can be received from the server 500, and the operation control signal information can be transmitted to the server 500. The base station communication method is W-CDMA (Wideband Code Division Multiple Access). de Division Multiple Access) and GSM (Global System for Mobile Long-distance wireless communication such as wireless communications can be used.

[0043] The main body side short-range wireless communication device 132 is connected to the first wearable device within a range where short-range wireless communication is possible. The wearable terminal 200, the second wearable terminal 300, and the input controller 400 are connected to each other. The main body side short-range wireless communication device 132 is a communication interface for performing short-range wireless communication. For example, the information is acquired by using an electronic tag, but is not limited to this. When the terminal 200, the second wearable terminal 300, and the input controller 400 are near each other, If it is at least capable of wireless communication, Bluetooth (registered trademark), IrD A (Infrared Data Association), Zigbee (registered trademark), HomeRF (Home Ra io Frequency, registered trademark), or wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.1 1g) may be used.

[0044] The first wearable device 200 includes a heart rate sensor 201, a first vibrator 202, a first proximity sensor 203, a second proximity sensor 204, a first proximity sensor 205, a first proximity sensor 206, a first proximity sensor 207, a first proximity sensor 208, a first proximity sensor 209, a first proximity sensor 210, a first proximity sensor 211, a first proximity sensor 212, a first proximity sensor 213 The heart rate sensor 201 includes a distance wireless communication device 203 (corresponding to a terminal side communication device). The device is worn closely around the user's chest and accurately detects the user's heart rate. The receiver 203 transmits the detected heart rate information to the HMD 100 via short-range wireless communication. The MD100 receives the transmitted heart rate information via the main body short-range wireless communication device 132. Captured into HMD100.

[0045] The first vibrator 202 generates vibrations in response to a vibrator control signal (see FIG. 3). The device is attached closely to the user's chest, ensuring that the generated vibrations are transmitted to the user. Notification information sent from the HMD100 to the user can be transmitted to the main unit by a nearby wireless The signal is transmitted to the first vibrator 202 via the line communication device 132 and the first short-range wireless communication device 203. The first vibrator 202 converts the notification information into vibration to notify the user. can.

[0046] The second wearable device 300 includes a blood pressure sensor 301, a second vibrator 302, a second proximal The blood pressure sensor 301 has a long-distance wireless communication device 303 (corresponding to a terminal-side communication device). The device is worn around the user's arm and detects the user's blood pressure with high accuracy, and communicates with the user via a second short-range wireless communication. The blood pressure information detected by the device 303 is transmitted to the HMD 100 by short-range wireless communication. In 100, the transmitted blood pressure information is received by the main body side short-range wireless communication device 132, and Capture it within 00.

[0047] The second vibrator 302 generates vibrations in response to a vibrator control signal (see FIG. 3). By wrapping it around the user's arm, the generated vibrations are transmitted to the user reliably. Notification information sent from the HMD100 to the user can be transmitted to the device in close proximity. The wireless communication device 132 transmits the signal to the second vibrator 302 via the second short-range wireless communication device 303. The second vibrator 302 converts the notification information into vibration to notify the user. can.

[0048] The main body side vibrator 124, the first vibrator 202, and the second vibrator 302 So, if there is no abnormal condition in the user's body or periphery, the virtual space information can be superimposed or swapped. When an action is started or when there is an abnormal condition in the user's body or surroundings, the virtual space information becomes heavy. When the tatami display or swap display operation is not started, the start and operation of the virtual space information display A vibration is generated to notify the user that the start is impossible. A sound is output from the headphones 122 to notify the user of the start or the impossibility of the display operation. and notify the user.

[0049] The input controller 400 includes an input operation device 401 and a third short-range wireless communication device 402. The input operation device 401 is an operation member such as a keyboard or key buttons, and is used by the user. You can set and input the information you want to enter.

[0050] Alternatively, the input operation device 401 may be formed within the display screen of the display 119. For example, in the case of a touchpad-type input device such as a capacitance type, the device is operated by approaching or The input operation unit 401 detects a touch operation as an operation input. , is transmitted to the HMD 100 via the third short-range wireless communication device 402, and The signal is received by the short-range wireless communication device 132 and captured in the HMD 100. The wearable terminal 200, the second wearable terminal 300, and the input controller 400 each include: This shows the case where information is easily transmitted and received via short-range wireless communication with the HMD100. However, a wired connection is also possible.

[0051] The server 500 includes an image processor 501 (a virtual space information generating processing unit), a server-side memory memory 502, server-side controller 503, server-side network communicator 504, antenna 50 5, which are interconnected via a bus 506.

[0052] The image processor 501 is a virtual machine that expresses a virtual space that is different from the real space with images and sounds. Generates imaginary space information.

[0053] The server-side memory 502 is a flash memory or the like, and is The generated virtual space information and various programs used by the server-side controller 503 in the server 500 I remember Ram and others.

[0054] The server-side network communicator 504 communicates with the external network 60 via the antenna 505. It is a communication interface that communicates with the HMD 100 via and sends and receives information.

[0055] The server-side controller 503 is configured with a processor such as a CPU or MPU, and 502, the OS (Operating System) and the operation control application, etc. By executing the program 126, each component is controlled, and the entire server 500 The server-side controller 503 performs operation control processing, similar to the main-body-side controller 125. It may be configured as a kit.

[0056] In response to a request from the HMD 100, the server-side controller 503 controls the image processor 50 The virtual space information generated in 1 or the virtual space information stored in the server-side memory 502 The controller 102 controls the operation of each component so as to transmit and supply information to the HMD 100.

[0057] 3 is a functional block diagram showing the functions of the main body controller 125. The figure roughly includes a surroundings monitoring determination unit 1251, a state determination unit 1252, and a display control unit 1253. The details of the processing of each component will be explained below with reference to Figures 4 and 5.

[0058] FIG. 4 is a flowchart showing the processing flow of the HMD system 1. 12 is a diagram showing the types of display modes determined by the control unit 1253. FIG.

[0059] As shown in FIG. 4, when the main power of the HMD system 1 is turned on, the surroundings monitoring process (S101 , S102) and the user status monitoring process (S201, S202) start in parallel.

[0060] Specifically, the surroundings monitoring determination unit 1251 monitors the real space information captured by the camera 111 and the surroundings. The monitoring sensor 118 acquires a first sensor output indicating the position and movement of the surrounding object detected (S1 01) The user's peripheral state is dangerous to the user when displaying virtual space information. The surroundings monitoring determination unit 1251 determines whether or not there is an abnormal state (S102). The monitoring and determination results are output to the display control unit 1253.

[0061] On the other hand, the state determination unit 1252 determines whether the first wearable terminal 200 and the second wearable terminal The second sensor output is acquired from each of the sensors 300 (S201), and the user's physical state is reflected in the virtual space information. It is determined whether or not there is an abnormal state in which the information is mistakenly recognized as real space information (S202). "The user's physical condition is abnormal" means that the user misinterprets the real space information and the virtual space information. It refers to a state in which attention is reduced to the extent that it is felt as if one is suffering from a cold or a broken bone, or other illness or symptom. isn't it.

[0062] The user's physical condition is abnormal if, for example, the user's blood pressure is within a predetermined normal blood pressure range. If you are not feeling well, have low or high blood pressure, or your heart rate is not within the normal range and is slow If the temperature is too high or too low, it is determined that there is something wrong with the user's physical condition. The determining unit 1252 outputs the result of the user's physical condition determination to the display control unit 1253.

[0063] The display control unit 1253 uses both the surroundings monitoring determination result and the user's physical condition determination result. The display format of the virtual space information is determined based on the

[0064] As shown in FIG. 5, the display control unit 1253 displays a screen image when the user's physical condition is normal and the surroundings are normal. In this case (status 1), the normal display mode, that is, the display mode of the virtual space information is set to the HMD 100. In other words, in the normal display mode, the image displayed to the user is displayed with the default image quality and color tone. No special effort is required to distinguish between real-world images and scenery and virtual space information. It displays virtual space information.

[0065] When the user's physical condition is abnormal but the surroundings are normal (status 2), or when the user's physical condition is abnormal, If the body condition is normal but there is an abnormality in the surrounding area that requires attention (status 3), a specific display state will be displayed. Display as follows.

[0066] The specific display mode is a display mode that presents virtual space information in a manner that is clearly different from real space information and has a lowered level of reality. In this way, the information that has started to be displayed to the user is virtual space information. This makes it possible to clearly recognize that

[0067] For example, the color, brightness, contrast, and resolution of the virtual space information to be superimposed or displayed interchangeably. The image quality of the real space information captured by the camera 111 and detected is clearly the same as the image quality of the real space information. In other words, the image processor 501 in the server 500 or the HMD 100 The virtual space information stored in the memory 128 is controlled and adjusted by the main body controller 125 .

[0068] In addition, a transparent mode may be used to display virtual space information, and when expressing the curved surface of an object, The number of polygons used for highlighting the outline can be reduced, or In addition, a message indicating that the information is virtual space information may be displayed on the image 13 of the virtual space information. It may be displayed in parallel with 0. Furthermore, it may be displayed in a virtual color different from the display color in the normal display mode. The space information may be displayed, or the virtual space information may be displayed in a blinking manner.

[0069] The specific display mode is not only at the start of displaying the virtual space information, but also continues while the virtual space information is being displayed. In this way, the display reality of the virtual space information can be reduced and the virtual space can be displayed. It is easy to understand for the user to understand the start of the virtual space information display and the display itself. This can make people aware that it is information.

[0070] When the user's physical condition is abnormal and there is an abnormality in the surrounding area (status 4) The virtual space information is temporarily hidden. Then, the virtual space information is hidden for a predetermined waiting time T th Will the user's physical condition recover and return to normal within a certain time (transition from status 4 to status 3)? When the surroundings become normal (transition from status 4 to status 2), the virtual Resume displaying the virtual space information. The time T elapsed since status 4 was reached is the waiting time for display. T th After this time, the display of virtual space information will be forcibly stopped.

[0071] If there is an abnormality in the surrounding area (status 5), the user's physical condition is normal or Regardless of the abnormality, the display of virtual space information is forcibly stopped.

[0072] Returning to FIG. 4, the main body controller 125 receives the surroundings monitoring determination result and the user's physical condition determination result. The following shows an example of how to determine the display mode. The procedure for determining which display mode to use is not limited to the following.

[0073] The main body controller 125 determines that the surrounding environment is at an alert level based on the surrounding monitoring determination result. When the user presses the button (S301), the display control unit 1253 determines to forcibly stop the display of the virtual space information. The virtual space information is forcibly stopped (S302: Status 5 in FIG. 5), and step S312 Proceed to.

[0074] When the virtual space information is to be displayed again, the user operates the input operation device 40 of the input controller 400. Enter the redisplay input instructions from 1. To make the redisplay input instructions easier to operate, At step S302, the output of the virtual space information is stopped and the redisplay icon is displayed on the display 119. When the user touches this icon, the In this case, the operations of steps S101 and S201 may be restarted.

[0075] The main body controller 125 determines that the surrounding environment is at a caution level based on the surrounding monitoring determination result. When the user is notified (S301 / No, S303 / Yes), the user's physical information is displayed based on the result of the physical information determination. It is determined whether the physical condition is normal or abnormal (S304).

[0076] If the user's physical information is normal (S304 / Yes), the display mode of the virtual space information is The display control unit 1253 determines to use the specific display mode and outputs an instruction to use the specific display mode. The display control unit 1253 converts the virtual space information into display data in a specific display mode, and The result is output to the display 119 (S305: Status 3 in FIG. 5).

[0077] If the user's physical information is abnormal (S304 / No), the display of the virtual space information is temporarily stopped. (S306: Status 4 in FIG. 5), and then temporarily stopped by the timer 133. The elapsed time T is measured (S307). th Not yet If it is not satisfied (S308 / Yes), the process returns to steps S101 and S201. As a result, the elapsed time T is the non-display waiting time T th If this occurs (S308 / No), the device will be forced to stop. (S309), and the process proceeds to step S312.

[0078] When the main body controller 125 determines that the surrounding environment is normal based on the surrounding monitoring determination result ( S303 / No), and determines whether the user's physical condition is normal or abnormal based on the result of the user's physical information determination. It is determined whether the current state is normal (S310).

[0079] If the user's physical information is normal (S310 / Yes), the display mode of the virtual space information is normal. The display control unit 1253 determines to switch to the normal display mode (S311: Status 1 in FIG. 5). The display control unit 1253 outputs an instruction to use the normal display mode. The data is converted into display data according to the user's preference and output to the display 119 (S311).

[0080] If the user's physical information is abnormal (S310 / No), the display mode of the virtual space information is It is decided to set the display mode (S305: Status 2 in FIG. 5).

[0081] The forced stop in steps S302 and S309 and the provisional stop in step S306 If you choose not to display the virtual space, a message will be displayed indicating that the virtual space information cannot be displayed. The notification may be displayed on the headphone 122. The voice to the user, the main body side vibrator 124, the first vibrator 202, the second vibrator The notification may be provided to the user by tactile vibrations from at least one of the vibrators 302 .

[0082] In addition, the virtual space in the specific display mode of step S305 or the normal display mode of step S311 When displaying virtual space information, a message indicating that virtual space information will be displayed is also displayed. The notification may be displayed on the headphone 119 to notify the user. 22 to the user, the main body side vibrator 124, the first vibrator 202, The notification may be provided to the user by a tactile vibration from at least one of the second vibrators 302. stomach.

[0083] The following will determine whether there are any factors that reduce attention due to the user's physical condition and the surrounding environment. An example will be described.

[0084] <Example of determining the user's physical state: saccade detection> The sensor device used to determine the user's physical condition in steps S201 and S202 An example in which the right eye gaze sensor 112 and the left eye gaze sensor 113 are used will now be described.

[0085] The right eye gaze data output by the right eye gaze sensor 112 and the left eye gaze data output by the left eye gaze sensor 113 are The main body controller 125 acquires the left eye gaze data and detects the movement of both eyes.

[0086] The state determination unit 1252 of the main body side controller 125 determines whether the right eye gaze data and the left eye gaze data are correct or not. Also, check whether the eyes are moving rapidly (saccades) If the user is making rapid eye movements, the system distinguishes between virtual space information and real space information. This is judged to be an abnormal state in which the information is mistakenly recognized as the actual information.

[0087] <Example of determining the user's physical state: Detecting whether the user is gazing at an object in real space> Another example of using gaze detection is to determine whether a user is gazing at a specific object in real space. There is a method to judge the degree of attention to the surroundings by observing the gaze. This can be defined as a state in which the gaze is directed at an object. Fig. 7 shows the coordinate system when the eye is looking at the object O. This is a diagram.

[0088] First, the state determination unit 1252 acquires real space information using the camera 111. The determination unit 1252 detects the user's gaze based on the right eye gaze data and left eye gaze data. The mounting position of the camera 111 on the HMD 100 is fixed, and the user By wearing the camera 111, the relative position between the eyeball and the camera 111 is fixed. is placed near the midpoint of the line connecting both eyes. The real space is viewed from the camera 111 in the direction (θ C , φ C ) coordinate system (Fig. 6) The point in the image of this real space where the user's gaze is directed is determined by the right eye gaze data. and the left eye gaze data can be calculated as follows:

[0089] In Figure 7, R represents the position of the right eye, and L represents the position of the left eye. The distance between the two eyes is S. C is This is the position of the camera 111, which is the midpoint of the baseline LR connecting both eyes. The position of the object will deviate from the midpoint of the baseline, but the position of the object is usually smaller than the amount of this deviation. Since it is far enough away, when determining the direction of the subject as seen from the camera 111, The error can be ignored.

[0090] When considering the distance and direction to the subject, the line formed by the baseline LR and the optical axis of the camera 111 Let us consider a reference plane P. The angle between the vector from each point to the subject and this reference plane P is expressed as θ. The direction above the reference plane P is taken as positive. Also, the vector from each point to the subject is projected onto the reference plane P. The angle formed by the projected vector and the optical axis direction of the camera 111 from each point is represented by φ. When viewed from the right, the angle to the right is taken as positive.

[0091] The left and right gaze sensors detect the direction of the subject as seen from each eye (θ L ,φ L ) and ( θ R ,φ R ) can be measured. From this data, the object O being watched by the camera 111 can be calculated. direction (θ C ,φ C ) and the distance d from the camera 111 to the subject O CO can be calculated. The specific formula is as follows:

number

number

number

[0092] In addition, since the distance to the gaze point can be known, the camera 111 also has a distance measurement function, and the distance to the subject can be measured. If the distance is also known, it is possible to more accurately determine whether the object is being gazed at (the camera's Other distance measuring devices may also be provided.

[0093] When the user's gaze point remains on the same object in the real world for a predetermined period of time or more , and it is determined that the driver is staring at the object and is not paying attention to other surrounding circumstances. When the subject is not gazing at any object, the physical state is judged as abnormal. It may be determined.

[0094] This allows the user to see the real-space information without having to worry about the virtual-space information display. There is no disturbance or interference, making the HMD100 safe and easy to use.

[0095] <Example of determining the user's physical state: Detecting whether the user has their eyes closed> As another example using gaze detection, the state determination unit 1252 detects right eye gaze data and left eye gaze data. Using data, it detects the movement of both eyes and determines whether the eyes are closed or not, other than by blinking. Good too.

[0096] The state determination unit 1252 determines the physical state of the user when the eyes are closed for reasons other than blinking. If the user's eyes are not closed except by blinking, the system determines that the user's physical condition is abnormal, especially at the alert level. The state may be determined to be normal.

[0097] This allows the virtual space information to be displayed even when the eyes are closed and the user is not concentrating, except for when blinking. This can increase the safety of the user by not displaying the message. The closed state of the eyepiece may be detected by using the camera 111 instead of detecting the line of sight.

[0098] <Example of determining the user's physical state: Detecting eye movement> Also, if there is little movement of the eyes, the user may feel sleepy and be unable to see clearly. Therefore, the state determination unit 12 52, if the eye movement is small and slow within a given time, the user's consciousness is absent-minded. Since it cannot be denied that the user may be in a state of abnormality, especially in the alert level, It may be determined that:

[0099] This allows the user to see the state of their body when they are drunk or unwell, for example. If the person is not fully conscious, the image 130 of the virtual space information is not displayed. This can improve user safety.

[0100] Based on the detected eye movements, the presence or absence of an abnormal state of the user's body is determined and the virtual space information is We have explained how to control the display behavior of virtual space information. The right eye line of sight sensor 112 and the left eye line of sight sensor 113 detect the movement of the eyeballs, and the eyeballs move in the virtual space. It identifies whether or not the eyes are moving in a fixed or tracking manner toward the information, and then interprets the virtual space information. The user did not make the movements that were captured, and it was determined that the user was not aware of or conscious of the virtual space information. In this case, the display of virtual space information will be temporarily hidden (status 4) or forcibly stopped (status It may be treated as status 5).

[0101] In this case, the displayed information is sufficient for the user to only be the real space information that they are aware of. This eliminates the display of unconscious virtual space information, allowing for safer head-mounted device use. This is expected to have the effect of allowing users to use the display.

[0102] <Example of detecting the user's physical state: head movement> In steps S201 and S202, a sensor device detects the user's physical condition. The acceleration sensor 114, the gyro sensor 115, and the geomagnetic sensor 116 are used as The user's head movement may be detected to detect the user's physical state.

[0103] In step S201, the state determination unit 1252 detects the acceleration sensor 114, the gyro sensor The sensor outputs output from the geomagnetic sensor 115 and the geomagnetic sensor 116 are acquired.

[0104] In step S202, the state determination unit 1252 determines whether the head is moving significantly. If the head is moving significantly, at least one of the outputs of each sensor is detected. The first is the motion determination threshold (acceleration threshold, angular velocity threshold) determined for each type of sensor output. If the threshold for the geomagnetic field and the threshold for the geomagnetic field change are exceeded, the user's physical condition will be affected by the virtual space information. This is determined to be an abnormal state in which the visual information is mistakenly recognized as real space information.

[0105] On the other hand, the state determination unit 1252 determines whether the head is not moving significantly, specifically, whether the outputs of the sensors are If all of these are equal to or less than the movement determination threshold, it is determined that there is no abnormality in the physical condition.

[0106] <Example of detecting the user's physical state: Example using voice information> In step S201, the state determination unit 1252 detects the surrounding sound microphone 120 and the vocal sound microphone 121. In addition, the state determination unit 1252 detects the sound by the right eye gaze sensor 112 and the left eye gaze sensor 113. The eye movement is captured using the eye gaze sensor 113.

[0107] In step S202, the state determination unit 1252 determines whether the user is in a state where the user is not present based on the captured voice and eye movement. It identifies where the user's consciousness is, and detects when the user is talking to someone or on the phone. If it is determined that the subject is viewing information that is different from the virtual space information, it is determined that there is something wrong with the subject's physical condition. On the other hand, if it is determined that the person is only viewing virtual space information, the physical condition is normal. It is determined that

[0108] <Example of detecting surrounding environment: detecting approaching objects> The surroundings monitoring sensor 118 is used as a sensor device for detecting the state of the surrounding environment of the user. This section explains the cases where

[0109] The surroundings monitoring sensor 118 detects the distance and direction of objects around the user. In step S101, the surroundings monitoring sensor 118 detects the distance and direction to an object present around the user. The surroundings monitoring determination unit 1251 acquires the detected and output first sensor output.

[0110] In step S202, the surroundings monitoring determination unit 1251 receives the first sensor from the surroundings monitoring sensor 118. Based on the sensor output, the system detects when an object such as a car, person, or animal approaches within a certain range. If an object is approaching within a certain range, the user The state of the periphery of the virtual space is dangerous to the user when displaying the virtual space information. If it is determined that an object is present within a predetermined distance range, it is determined that there is something abnormal in the surrounding environment. If there is no approaching object within the distance range, the surrounding environment is judged to be normal. If so, it may be judged as a particularly alert level.

[0111] This allows the system to identify and display virtual space information when an object is approaching within the user's distance range. The display can be set to a fixed mode or stopped, and the virtual space can be used in a dangerous environment for the user. This prevents inconvenience caused by displaying information. This allows you to notify users that something is wrong with the host environment.

[0112] <Example of detecting surrounding environment: detecting approaching objects> A temperature and humidity sensor 117 is used as a sensor device to detect the user's surrounding conditions. The temperature and humidity around the laser may also be detected.

[0113] In step S101, the surroundings monitoring determination unit 1251 determines whether the temperature and humidity detected by the temperature and humidity sensor 117 is The temperature and humidity are detected and output from the sensor. The data is temporarily stored in the memory.

[0114] In step S102, the surroundings monitoring determination unit 1251 determines whether the surroundings monitoring determination unit 1251 detects the temperature and humidity stored in the memory 128. The temperature change is calculated and the temperature change exceeds a predetermined temperature change threshold or the humidity change is predicted. When at least one of the events occurs that exceeds the humidity change threshold set for the user, On the other hand, if the temperature change is below the temperature change threshold and the humidity change is If the humidity change is equal to or less than a predetermined humidity change threshold, the user's surrounding environment is determined to be normal.

[0115] In addition to temperature and humidity, it can also detect the user's surrounding conditions, such as air pressure and ambient noise. If the information is relevant to the user, the sensor for that information, for example, the atmospheric pressure sensor or the ambient sound microphone 120, can be used. The sensor 118 may be used as a surroundings monitoring sensor.

[0116] The effects of the HMD system 1 according to this embodiment will be described. Whether or not the information is mistaken for real-world information and the degree of inconvenience caused by such a misrecognition depend on the user's judgment. It depends on the combination of your physical condition and the surrounding environment.

[0117] Generally, if the user's physical condition is normal, their attention level is high, so they are less likely to make mistakes in the virtual space. It is thought that recognition is unlikely to occur.

[0118] In addition, the user's physical condition was abnormal, and as a result, their attention was reduced, resulting in misrecognition of virtual space information. However, if the surrounding environment is normal and safe, for example, sitting on a sofa indoors with a computer on the table, The cup is displayed as virtual space information, and there is no problem if it is mistaken for a real cup. On the other hand, if you misinterpret virtual space information while walking outdoors, you may stumble or fall onto the ground. This can cause serious problems, such as interference with objects.

[0119] Furthermore, regardless of the user's physical condition, for example, when walking outdoors, a vehicle or bicycle may come into contact with the user. When driving towards the destination, the user can quickly move away from the situation where they are distracted by virtual space information. It would be better to let the

[0120] According to the HMD system 1 of this embodiment, the normal state of the user's body and the surrounding environment can be monitored. The display mode of virtual space information is determined by combining both the physical condition of the user and the abnormality. Compared to focusing only on the virtual space, it is possible to suppress the occurrence of misrecognition of virtual space information, and HMD1 00 will be easier to use.

[0121] In addition, the state of the surrounding environment can be divided into caution and alert levels, allowing you to know when there is something abnormal in the surrounding environment. The virtual space information is not immediately hidden when the image 130 of the virtual space information is displayed. It is possible to balance enjoying the experience with responding to abnormalities in the surrounding environment.

[0122] In addition, if the user's physical condition is abnormal and there is an abnormality in the surrounding environment, In status 4), the display of virtual space information is temporarily stopped. If the bell abnormality is resolved within the non-display waiting time, the virtual space information will be displayed again in the specified display mode. Abnormalities in attention level may be resolved in a short period of time. The display of virtual space information will resume, improving the usability of the HMD100.

[0123] The present invention is not limited to the above-described embodiment, and includes various modifications. The embodiments described above have been described in detail to clearly explain the present invention, and are not necessarily intended to be limiting. The present invention is not limited to the configurations described in the above. It is possible to replace a part of the configuration with the configuration of another embodiment, and the configuration of one embodiment may be replaced with the configuration of another embodiment. It is also possible to add the configuration of other embodiments to the configuration of each embodiment. It is possible to add, remove, or replace other configurations.

[0124] For example, in the above example, the image processor 501 that generates the virtual space information is installed in the server 500. The virtual space information is received via the external network 600 and displayed. The image processor 501 may be incorporated into the HMD 100. Without the need for communication, virtual space information or modified virtual space information can be transmitted to the It may be displayed in a display mode.

[0125] The heart rate sensor 201 in the first wearable device 200 and the heart rate sensor 202 in the second wearable device 300 The blood pressure sensor 301 in the HMD 100 and the input operation unit 401 in the input controller 400 are In this case, the heart rate sensor 201 is attached closely to the head and measures the heart rate. The blood pressure sensor 301 is also attached closely to the head and detects the blood pressure value in the cephalic artery just below the scalp. The input operation device 401 is set in a position in the HMD 100 where the user can easily perform input operations. That's fine.

[0126] Alternatively, the user may utter a voice indicating an input operation, and the voice may be collected by the voice microphone 121 to indicate the input operation. Production information may also be captured.

[0127] Also, an input operation screen is displayed on the display 119, and the right eye gaze data and the left eye gaze data are input. The input operation information is captured based on the position on the input operation screen where the gaze is directed, as detected by the Alternatively, a pointer may be displayed on the input operation screen and the input operation may be performed by specifying the pointer with a hand movement or the like. By using voice input and display, usability can be further improved. It is possible to improve it.

[0128] In this way, the first wearable terminal 200, the second wearable terminal 300, the input controller The image processor 501 may be integrated into the HMD 100. It may also be incorporated into

[0129] In addition, other biological sensors such as sweat sensors and breathing sensors are used to detect the user's physical condition. A sensor for acquiring body information may be used.

[0130] As mentioned above, various sensor devices are used to identify and judge the user's physical state or surrounding conditions, and The operation for controlling the display of spatial information has been explained. When the display control unit 1253 determines that the user's eyes are in a dark state, the display control unit 1253 applies strong light to the user's eyes. Or, two or more blinking lights are displayed in the user's field of view, and the surroundings are monitored again. Based on the determination results from the state determination unit 1251 and the state determination unit 1252, the brain is awakened and consciousness is clear. After transitioning to the corresponding awakening state (status 1, 2), virtual space information is superimposed, Alternatively, the display may be switched.

[0131] In addition, users value the immersive feeling of playing games and watching movies, and use the HMD100. In this case, the display of virtual space information is controlled according to the user's physical condition or the state of the surrounding environment. In this case, for example, the input controller 400 may not be used to input virtual space information. The user may input a setting as to whether or not to perform display control.

[0132] The above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in part or in whole by, for example, an integrated circuit. The above configurations, functions, etc. may be realized by hardware by designing them in advance. The processor interprets and executes the program 126 that realizes each function, The functions may be realized by software. Such information is stored in memory, hard disks, SSDs (Solid State Drives) and other storage devices, Alternatively, it can be placed on a recording medium such as an IC card, an SD card, or a DVD.

[0133] In addition, control lines and information lines are shown as they are considered necessary for explanation purposes, and do not necessarily represent all the lines in the product. This does not necessarily show all control and information lines. In reality, almost all components are interconnected. It can be said that this is the case. [Explanation of symbols]

[0134] 1: HMD system 100: HMD 111: Camera 112: Right eye gaze sensor 113: Left eye gaze sensor 114: Acceleration sensor 115: Gyro sensor 116: Geomagnetic sensor 117: Temperature and humidity sensor 118: Surrounding area monitoring sensor 119: Display 120: Peripheral sound microphone 121: Speech microphone 122: Headphones 124: Main unit vibrator 125: Main unit controller 126: Program 127: Information Data 128: Memory 129: Antenna 130:Video 131: Main unit network communication device 132: Main unit short-range wireless communication device 133: Timer 140: Bus 200: First wearable device 300: Second wearable device 400: Input controller 500: Server 600: External network

Claims

1. A head-mounted information processing device, A display unit; a camera that captures an image of a real space in front of the head-mounted information processing device; a gaze sensor for detecting a gaze of a user of the head-mounted information processing device; A control unit; Equipped with The control unit Generate virtual objects, displaying the generated virtual object and a real object included in the real space photographed by the camera on the display unit; determining whether the user's gaze remains on the real object displayed on the display unit for a predetermined period of time or longer based on the detection result of the gaze sensor and whether the user is gazing at the real object; When it is determined that the user is not gazing at the real object, the virtual object is displayed in a normal display mode; When it is determined that the user is gazing at the real object, the display unit is instructed to display the virtual object in a specific display mode or to suspend display. Head-mounted information processing device.

2. The head-mounted information processing device according to claim 1, a communication unit that is communicatively connected to an input controller that accepts instructions from the user; the control unit, after instructing to suspend the display of the virtual object, resumes the display of the virtual object when receiving a re-display instruction from the user from the input controller via the communication unit. Head-mounted information processing device.

3. 3. The head-mounted information processing device according to claim 1, The display in the specific display mode includes at least one of displaying the virtual object at a resolution lower than the resolution in the normal display mode, displaying the virtual object in a display color different from the display color in the normal display mode, blinking the virtual object, highlighting the outline of the virtual object, and displaying the virtual object and a message indicating that it is the virtual object side by side. Head-mounted information processing device.

4. 3. The head-mounted information processing device according to claim 1, When instructing the display unit to suspend display, the control unit causes the display unit to display a message indicating that the virtual object cannot be displayed. Head-mounted information processing device.

5. The head-mounted information processing device according to claim 1 or 2, further comprising an audio output unit, When instructing the display unit to suspend display, the control unit causes the audio output unit to output an audio notification indicating that the virtual object cannot be displayed. Head-mounted information processing device.

6. The head-mounted information processing device according to claim 1 or 2, Further comprising a vibrator, When instructing the display unit to suspend display, the control unit causes the vibrator to output a vibration notification indicating that the virtual object cannot be displayed. Head-mounted information processing device.

7. A head-mounted information processing device according to claim 1 or 2, When the control unit determines that the user is gazing at the real object, the control unit controls the display unit to display the virtual object in the specific display mode or to stop displaying the virtual object so as to suppress disturbance or obstruction caused by the virtual object and increase safety. Head-mounted information processing device.

Images