Information processing device and information processing method

Abstract

An information processing device comprises: a processor; a line-of-sight sensor that detects the line-of-sight direction of a user of the information processing device and outputs line-of-sight direction information; an obstacle sensor that detects a surrounding obstacle in the travel direction of the user and outputs obstacle information; and a display interface. The processor measures, on the basis of the line-of-sight direction information acquired from the line-of-sight sensor, a staying time during which the line-of-sight direction of the user stays within a predetermined staying determination range. When it is assessed that the measured staying time is equal to or greater than a threshold value, the processor causes the display interface to display, on the basis of the obstacle information acquired from the obstacle sensor, a virtual object that is to be displayed so as to be superimposed on the obstacle or is to be displayed close to the obstacle.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present invention particularly relates to an information processing apparatus and an information processing method suitable for driving support.

[0002] In recent years, Augmented Reality (AR) technology that overlays a virtual object (virtual object) on a virtual space composed of digital information created by, for example, Computer Graphics (CG) onto the real space to visually expand the real world has been widely used, and information processing apparatuses that can easily visually recognize both real-space objects and virtual objects while recognizing real-space objects have become widespread. As an example, a Head Mounted Display (HMD) worn on the head, and AR glasses, smart helmets, and smart visors, which are digital devices in the form of glasses or helmets as forms of HMDs, can be mentioned. In HMDs, AR glasses, smart helmets, and smart visors, while displaying a virtual object by a display interface and allowing the user to visually recognize a real object in front of the eyes (so-called optical see-through type), or displaying a real object or virtual object in the real space on a display provided on the front of the head (so-called video see-through type), an experience as if a virtual object exists on the spot in the real world can be provided. The virtual object is generated from digital information composed of various types of display items such as images, illustrations, and text characters, and is visualized by being displayed in front of the user's eyes.

[0003] On the other hand, as a vehicle driving support technology, Patent Document 1 describes, "Detect the line-of-sight position of a driver during driving, and for example, create a line-of-sight map representing the frequency distribution of the line-of-sight direction in the past 5 seconds. Calculate the sum of squared differences between the created line-of-sight map and a line-of-sight map with a sound line-of-sight distribution, and if this sum of squared differences is a value equal to or greater than a predetermined value, estimate that the driver is not making a sound line-of-sight distribution and generate an alarm or the like. (Summary excerpt)".

[0004] Japanese Patent Application Laid-Open No. 10-244892

[0005] Vehicles that involve shifting the center of gravity, such as motorcycles, bicycles, electric scooters, and unicycles, are easily affected by road surface conditions while riding, so users tend to focus on the road surface. Also, while riding, the driver steers in the direction that is causing the vehicle to tip over, so the center of gravity periodically tilts from side to side, causing the vehicle to move in a zigzag pattern. Therefore, when attempting to apply the driving assistance technology described in Patent Document 1 to drivers of vehicles that involve shifting the center of gravity, there is a challenge in that they are relatively more affected by fatigue, road surface conditions, and cornering compared to drivers of four-wheeled vehicles, making it difficult to define a standardized "distribution of gaze direction in a healthy driving state."

[0006] This invention has been made in view of the above problems, and aims to provide a driving assistance technology suitable for drivers of vehicles, especially drivers of vehicles that involve shifting the center of gravity.

[0007] To solve the above problems, the present invention has the configuration described in the claims. To give an example, the present invention is an information processing device comprising a processor, a gaze sensor that detects the gaze direction of the user of the information processing device and outputs gaze direction information, an obstacle sensor that detects obstacles around the user's direction of travel and outputs obstacle information, and a display interface. Based on the gaze direction information obtained from the gaze sensor, the processor measures the dwell time during which the user's gaze direction remains within a predetermined dwell time determination range, and when it is determined that the measured dwell time exceeds a threshold, it displays a virtual object on the display interface that is superimposed on or close to the obstacle detected by the obstacle sensor, based on the obstacle information obtained from the obstacle sensor.

[0008] According to the present invention, it is possible to provide a driving assistance technology suitable for drivers of vehicles, particularly drivers of vehicles that involve shifts in the center of gravity. Other problems, configurations, and effects not mentioned above will be clarified by the following description of embodiments.

[0009] This is an explanatory diagram showing an overview of the HMD according to this embodiment. This is a functional block diagram of the HMD according to this embodiment. This is a diagram schematically illustrating the overview of the gaze dwell detection process. This is a diagram schematically illustrating the overview of the body movement detection process. This is a diagram illustrating a state in which the user's left-right movement period and tilt amplitude are regular (normal state). This is a diagram illustrating a state in which the user's left-right movement period and tilt amplitude are irregular (unstable state). This is a diagram showing an example of display when the user's gaze is directed towards obstacle information. This is a diagram showing an example of display when the user's gaze is not directed towards obstacle information. This is a flowchart showing the flow of obstacle notification processing using the results of the gaze dwell detection process. This is a flowchart showing the flow of obstacle notification processing using both gaze dwell detection processing and body movement detection processing. This is a flowchart showing the flow of obstacle notification processing using the results of body movement detection processing. This is a flowchart showing the flow of obstacle notification processing with added warning notification processing.

[0010] Embodiments of the present invention will be described below with reference to the drawings. Throughout the drawings, components and steps having the same function will be denoted by the same reference numerals, and repeated descriptions of them will be omitted.

[0011] The information processing device, information processing method, and information processing system according to this embodiment can provide effective driving assistance to drivers of vehicles, especially drivers of vehicles that involve shifts in the center of gravity. Therefore, since the present invention can enhance the commercial value of the information processing device to which the present invention is applied, it is expected to contribute to Sustainable Development Goal 8.2 (Increase economic productivity through diversification, technological advancement and innovation, particularly in industries that enhance the value of goods and services and in labor-intensive industries).

[0012] As a specific example of the information processing device according to this embodiment, we will explain using an HMD (Head-Mounted Display) that is worn on the head of a user riding a motorcycle, bicycle, balance scooter, electric kick scooter, or other two-wheeled vehicle, as well as a unicycle, tricycle, snowmobile, snowbike, jet ski, etc., allowing the user to see virtual objects such as driving instructions along with the physical objects in front of them. Furthermore, the present invention can be applied to information processing devices used by drivers of automobiles such as passenger cars and trucks to provide driving assistance, and it can also be adapted to situations such as walking that involves shifting the center of gravity.

[0013] Referring to Figures 1 and 2, an example configuration of the information processing device and information processing system according to this embodiment will be described. Figure 1 is an explanatory diagram showing an overview of the HMD according to this embodiment. Figure 2 is a functional block diagram of the HMD according to this embodiment.

[0014] In Figure 1, the user 10 is wearing an HMD 100 on their head. To avoid difficulty in viewing the diagram, the HMD 100 is also shown enlarged and separated from the user 10. The HMD 100 is equipped with a camera 101 that photographs the outside world around the user, a distance sensor 102 that measures the distance and angle to an object and captures the shape of the object in three dimensions, a left eye gaze sensor 103 and a right eye gaze sensor 104 that detect the lines of sight of the user 10's left and right eyes, a posture sensor 105 that captures the state of posture such as direction and tilt, a microphone 106 that collects sounds such as the user 10's voice, a speaker 107 that emits sound based on the voice data, a vibrator 108 that generates vibrations, a positioning sensor 109 that detects the current position, and a lamp 110 that emits light.

[0015] Furthermore, the HMD 100 communicates wirelessly with an external information server 192 (see Figure 2) via an external network 191 (see Figure 2), etc. The information server 192 contains information such as traffic navigation. The HMD 100 can receive and display traffic navigation information such as road conditions and congestion from the information server 192. In addition, the processing load on the HMD 100 may be reduced by having an information terminal 193 (see Figure 2) perform some of the processing of the HMD 100. In this way, the HMD 100, by being connected to at least one of the external devices, the information server 192 and the information terminal 193, constitutes an information processing system 1 suitable for driver assistance.

[0016] Here, if the user 10, wearing the HMD 100 on their head and riding the motorcycle 120, experiences a decrease in concentration and difficulty maintaining balance, a phenomenon occurs where the user 10's gaze in the up, down, left, and right directions continues to drift aimlessly towards a specific narrow field of view 121, as shown in Figure 1.

[0017] If person 122 suddenly jumps out from the alley in this situation, the driver's attention may not be focused on person 122, potentially leading to a delay in avoiding danger. On the other hand, if warnings are constantly issued for obstacles such as person 122, regardless of the driver's level of attention, there is a concern that drivers may become accustomed to the warnings, diminishing their effectiveness.

[0018] Therefore, the HMD100 does not issue obstacle notifications when it is presumed that the driver is concentrating on driving, and only issues obstacle notifications when it is presumed that the driver is not concentrating on driving.

[0019] As an example of how obstacles are notified, a frame object 131 surrounding a person 122 that has suddenly emerged from an alley, and an obstacle contour enhancement object 132 that emphasizes the contour of the person 122 are displayed on the display screen 130 as virtual objects (corresponding to a first virtual object). This allows the user 10 to visually perceive the virtual object that notifies the presence of the real person 122 together with the person 122. This is one of the features of the HMD 100 according to this embodiment.

[0020] As shown in Figure 2, the HMD 100 includes a camera 101, a distance sensor 102, a left eye gaze sensor 103, a right eye gaze sensor 104, a posture sensor 105, a microphone 106, a speaker 107, a vibrator 108, a positioning sensor 109, a lamp 110, an operation input interface 111, a network communication interface 112, a short-range wireless communication interface 113, a display interface 114, a processor 115, a memory 116, and a timer 117, with each component interconnected via a bus 118.

[0021] Camera 101 captures the user's field of view, converting light entering through the lens into an electrical signal using an image sensor to acquire an image. The HMD 100 detects obstacles around the user 10, particularly around the direction of travel, from this captured image. "Around the direction of travel" means the area in front (including the front and front left and right) when moving forward, and the area behind (including the rear and rear left and right) when moving backward.

[0022] The distance measuring sensor 102 is a sensor that can measure the distance and angle to an object and capture the shape of an object as a three-dimensional object. Distance measuring sensors 102 include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light and measures the scattered light that reflects back to analyze and detect the distance to an object at a distance, as well as the shape and state (whether it is moving, the direction of movement, etc.); TOF (Time Of Flight) sensors, which measure the distance by measuring the reflection time of pulsed light irradiated onto an object pixel by pixel; and millimeter-wave radar, which emits millimeter-wave radio waves and captures the reflected waves to detect the distance to the object reflecting the waves and the state of the object. In the HMD 100, the distance measuring sensor 102 is used to detect the distance to obstacles around the user 10.

[0023] The left eye gaze sensor 103 and the right eye gaze sensor 104 are sensors that detect the movement and orientation of the user's left and right eyes, respectively, to detect the user's gaze destination and direction. The process for detecting eye movement can be done using well-known techniques commonly used as eye tracking. For example, in a method using corneal reflection, an infrared LED (Light Emitting Diode) is shone on the face and captured by an infrared camera. The position of the reflected light on the cornea created by the infrared LED irradiation (corneal reflection) is used as a reference point, and the movement of the eye and gaze are detected based on the position of the pupil relative to the position of the corneal reflection. The HMD 100 can determine whether the gaze destination and direction of the user 10, as captured by the left eye gaze sensor 103 and the right eye gaze sensor 104, are directed towards obstacle information or the obstacle itself detected by the camera 101 and the distance measuring sensor 102.

[0024] The posture sensor 105 consists of an acceleration sensor 1051, a gyroscope sensor 1052, a geomagnetic sensor 1053, etc., and detects the orientation and tilt of the HMD 100. The acceleration sensor 1051 is a sensor that detects acceleration, which is the change in velocity per unit time, and can capture movement, vibration, shock, etc. The gyroscope sensor 1052 is a sensor that detects angular velocity in the rotational direction and can capture the vertical, horizontal, and diagonal orientation. The geomagnetic sensor 1053 is a sensor that detects the Earth's magnetic field and detects the direction the HMD 100 is facing. With these, the HMD 100 can detect the orientation and tilt of the user 10 and capture the lateral movement period and tilt amplitude of the user 10's body when riding a vehicle that involves a shift in the center of gravity.

[0025] The operation input interface 111 is an interface with input means such as gaze, hands, or a pointer, and information that the user 10 wants to input is set and input. As an example of the configuration of the operation input interface 111, for example, an input operation screen such as an operation menu may be displayed on the display screen of the display interface 114, and input operation information may be taken in by the position on the input operation screen where the gaze is directed, or a hand or pointer may be displayed on the input operation screen, and input operation information may be taken in by operating the hand or pointer. Examples of operation input include setting inputs for gaze calibration, motion period, tilt, and amplitude calibration, which are performed according to past driving conditions, conditions at the start of driving, conditions at regular intervals during driving, driving conditions above a certain speed, etc.

[0026] In the case of an optical see-through HMD, the display interface 114 includes a projector that projects virtual objects such as display items and notification information to the user, and a transparent half-mirror that forms an image of the projected virtual objects in front of the user's eyes. This allows the user 10 to see both the real objects in their field of view and the formed virtual objects together, as if they were floating in three dimensions. In the case of a video see-through HMD, the interface includes a display such as an LCD panel (corresponding to one form of the display interface 114) that displays both the real objects in front of the user's eyes, captured by the camera 101, and the virtual objects together. This allows the user 10 to see obstacle information consisting of virtual objects superimposed on the real objects in the field of view image in front of them.

[0027] The processor 115 consists of a CPU (Central Processing Unit), etc.

[0028] Memory 116 is composed of volatile memory such as RAM (Random Access Memory) and non-volatile memory such as ROM (Read Only Memory). The non-volatile memory stores the operating system (OS) 1162, application programs 1163 for operation control, and information data 1165. The information data 1165 stores gaze detection information 1166 indicating the detected gaze direction, gaze dwell time information 1167 indicating the dwell time of the gaze, motion period tilt amplitude information 1168 indicating the lateral movement period and tilt amplitude of the user's body, and obstacle information 1169 including frame objects surrounding obstacles and obstacle contour enhancement objects that highlight the contours of obstacles.

[0029] The processor 115 loads the operating system 1162 and application program 1163 into volatile memory and executes them, thereby enabling the functions of the OS, middleware, and applications, as well as the calibration processing unit 1151, gaze dwell determination unit 1152, body movement determination unit 1153, obstacle detection processing unit 1154, first virtual object processing unit 1155, and warning notification processing unit 1156, and the functions of each of these units are realized.

[0030] Microphone 106 collects the user's own voice and external sounds and converts them into audio data. When user 10 speaks to indicate instructions such as input operations, microphone 106 collects the sound, and HMD 100 takes in the instruction information, the user can easily perform actions in response to the instruction information.

[0031] The speaker 107 outputs sound based on audio data and can inform the user 10 of notification instructions by voice. For example, it can emit a loud warning sound like a siren or a warning message such as "Obstacle ahead!" to inform the user 10.

[0032] The vibrator 108 generates vibrations under control from the processor 115, and converts notification information transmitted to the user 10 by the HMD 100 into vibrations. The vibrator 108 transmits vibrations to the user's head wearing the HMD 100, allowing the user 10 to be notified of notification information, such as message guidance like "Obstacle ahead!", through vibration.

[0033] The positioning sensor 109 is a sensor that measures the current location of the HMD 100. For example, a positioning sensor using GNSS (Global Navigation Satellite System) may be used. The user 10 can know where they are driving by combining the measured current location of the user with traffic navigation information such as road conditions and congestion from an external information server 192, making it possible to drive to their destination comfortably without getting lost.

[0034] The lamp 110 emits light in response to the control of the processor 115, and can notify the user 10 of the presence of an obstacle by emitting light or flashing.

[0035] The network communication interface 112 is a communication interface that connects to the information server 192 via an external network 191 through base station communication or the like. As an example, the network communication interface 112 may be configured using a wireless communication interface for connecting to, for example, LTE (Long Term Evolution), a fourth-generation mobile communication system, a fifth-generation mobile communication system, or Wi-Fi (registered trademark).

[0036] The short-range wireless communication interface 113 is a communication interface that performs short-range wireless communication with an information terminal 193 carried by the user 10, such as a smartphone or tablet, and may be configured using wireless communication interfaces such as Bluetooth®, Wi-Fi®, Zigbee®, IrDA (Infrared Data Association®), HomeRF (Home Radio Frequency®), electronic tags, or wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g). The HMD 100 may be connected to an information server 192 via the short-range wireless communication interface 113 and the information terminal 193.

[0037] The calibration processing unit 1151, as part of the calibration of the line of sight direction, appropriately and optimally sets the setting values ​​of either the line of sight dwell time determination range 201, the line of sight dwell time threshold Tth, or both, according to past driving conditions, conditions at the start of driving, conditions at regular intervals during driving, driving conditions above a certain speed, etc. If the line of sight dwell time T during which the user 10's line of sight direction remains within the line of sight dwell time determination range 201 is greater than or equal to the line of sight dwell time threshold Tth, it is determined that the line of sight is dwelling. For example, the line of sight dwell time determination range 201 is defined by the angular displacement of the user 10's line of sight direction.

[0038] Furthermore, the calibration processing unit 1151 appropriately and optimally sets a predetermined stable operation determination range, which is defined by either the period of the user 10's left-right movement, the body tilt amplitude, or both, in order to determine that the user's body movement is stable, using past driving conditions, the condition at the start of driving, and recorded information of body movement collected at regular intervals during driving.

[0039] The calibration processing unit 1151 may perform calibration to optimize the gaze dwell time determination range 201 and the gaze dwell time threshold Tth as appropriate, according to the user 10's driving posture habits and the vehicle's stability at the start of driving and during driving. For example, the gaze dwell time determination range 201 and the gaze dwell time threshold Tth may be changed as appropriate based on the field of view angle when the user 10 has driven in the past, and set to values ​​optimized for each user 10. When setting the gaze dwell time determination range 201 and the gaze dwell time threshold Tth as appropriate for each user, the user may set the gaze dwell time determination range 201 and the gaze dwell time threshold Tth themselves before starting to drive, or the gaze dwell time determination range 201 and the gaze dwell time threshold Tth may be automatically set by taking a picture of the user 10's face with the camera 101 before putting on the HMD 100 and performing face recognition processing. This improves the accuracy of the determination by enabling gaze dwell time determination that matches each user's gaze movement, even if the way the eyes move differs from user to user.

[0040] As an example of calibration based on past driving conditions, the user 10 and gaze detection information 1166 indicating the movement of the user's gaze direction during past driving are associated and stored in memory 116, and either the gaze dwell time determination range, the gaze dwell time threshold, or a combination thereof (hereinafter referred to as "gaze dwell time determination conditions") are determined based on the past history of each user 10. Then, at the start of driving, the gaze dwell time determination conditions corresponding to the user 10 may be set by user authentication or user operation. Similarly, the period of the left-right movement of the user 10's body and the amplitude of the body tilt during past driving may be detected by the posture sensor 105, and a stable operation determination range corresponding to the user 10 may be set.

[0041] Furthermore, as an example of calibration according to the state at the start of driving, when user 10 inputs whether their physical condition is good or bad into the HMD 100, the calibration processing unit 1151 sets stricter gaze retention judgment conditions on days when the user is not feeling well compared to days when they are feeling well. As stricter gaze retention judgment conditions, for example, the field of view retention judgment range may be narrowed or the gaze retention time threshold may be shortened. Similarly, according to user 10's physical condition, on days when the user is not feeling well, stricter stable operation judgment ranges may be set, for example, by adjusting the length of the left-right movement period or by narrowing the body tilt amplitude.

[0042] Furthermore, as an example of calibration according to the state at regular intervals during driving, eye movements may differ depending on the degree of recognition of the driving route, for example, whether it is a familiar route or a new route. Therefore, the calibration processing unit 1151 may store location information measured in the past by the positioning sensor 109 and perform eye calibration at regular intervals by comparing the past location information with the current location. If the HMD 100 or information terminal 193 is equipped with a car navigation system, the calibration processing unit 1151 may compare the past actual driving route information recorded in the car navigation system with the changes in the location information of the positioning sensor 109 measured during a certain period of driving to determine whether the route the user 10 is driving is a familiar route or not. If the route the user 10 is traveling on is a route that has been driven for the first time, relatively strict eye-sight retention judgment conditions and stable operating range may be set. Also, if the driving time exceeds a certain period, for example, one hour, fatigue is expected to accumulate, so even stricter eye-sight retention judgment conditions and stable operating range may be set.

[0043] Furthermore, as an example of calibration corresponding to driving conditions above a certain speed, taking into account that as the speed increases, the vehicle enters stable driving mode and the user's gaze becomes fixed, stricter gaze retention judgment conditions and stable operation judgment ranges may be set when the user 10's driving speed exceeds a predetermined speed, for example, the speed limit.

[0044] Also, at the start of driving and when decelerating to a stop, the vehicle body is more likely to become unstable compared to when driving at a certain speed or higher. Therefore, based on the acceleration detected by the acceleration sensor 1051, the calibration processing unit 1151 determines the driving state of the vehicle body at the start of driving, during driving, when decelerating to a stop, etc., and performs calibration to optimize the line-of-sight retention determination range 201 and the line-of-sight retention time threshold Tth as appropriate according to each driving state. Further, the calibration processing unit 1151 may perform calibration to optimize the line-of-sight retention determination range 201 and the line-of-sight retention time threshold Tth as appropriate according to the acceleration measured by the acceleration sensor 1051, the time measured by the timer 117, or using a speed sensor (not shown) according to the speed of the vehicle body.

[0045] By performing line-of-sight calibration to optimally set the line-of-sight retention determination range 201 and the line-of-sight retention time threshold Tth according to the user's driving state in this way, it is possible to more optimally and accurately detect the measurement of the retention time in the line-of-sight direction and to more accurately detect a decrease in the user's attention.

[0046] Further, the calibration processing unit 1151 may perform movement period inclination amplitude calibration to appropriately and optimally set the determination criteria for the disturbance of the left-right movement period and inclination amplitude of the body according to the past driving state, the state at the start of driving, the state at each fixed period during driving, etc.

[0047] Since the way the body moves may change for each user 10 in the past driving state, it may be possible to calibrate the normal movement according to the user 10. Also, before driving, the user 10 may perform a setting operation or face authentication process by himself / herself, and the HMD 100 may determine the user 10 and call up past data.

[0048] Also, as the state at the start of driving, based on the physical condition of the day, etc., for example, when the physical condition is poor, the left-right movement period and inclination amplitude of the body may be set to be smaller compared to the past driving record.

[0049] Furthermore, the road conditions (unevenness, number of curves, narrowness) vary along the route taken, and even if the route is the same as in the past, the road surface conditions on that day may be dry, wet, or frozen, so they may not necessarily match past road conditions. Therefore, it is advisable to perform calibration at regular intervals during driving.

[0050] By performing calibration according to the user's riding condition in this way, it becomes possible to detect disturbances in the lateral movement cycle and tilt amplitude of the user's body with greater accuracy, making it easier and more accurate to detect a decrease in the user's attention.

[0051] The gaze dwell time determination unit 1152 measures the time during which the user's gaze detected by the left eye gaze sensor 103 and the right eye gaze sensor 104 is included in the gaze dwell time determination range 201 as the gaze dwell time, and determines whether or not the gaze dwell time threshold has been reached.

[0052] The body movement determination unit 1153 measures the lateral movement of the user 10's body detected by the posture sensor 105 and determines whether there is a disturbance in the movement period or tilt amplitude, which is an irregularity in the lateral movement period and tilt amplitude of the body.

[0053] The obstacle detection processing unit 1154 uses the camera 101 and the distance measuring sensor 102 to detect obstacles that would hinder travel along the direction of travel.

[0054] The first virtual object processing unit 1155 generates and displays a first virtual object that is superimposed on or displayed in close proximity to an obstacle. Examples of the first virtual object include a frame object that surrounds an obstacle and an obstacle contour enhancement object that emphasizes the outline of the obstacle.

[0055] The warning notification processing unit 1156 notifies the user of the presence of an obstacle in the direction of travel by outputting a warning sound or voice message from the speaker 107, vibrating with the vibrator 108, or flashing or lighting the lamp 110. It may also be equipped with a buzzer (not shown). The buzzer and speaker correspond to a warning sound emitter. The warning sound emitter, vibrator 108, and lamp 110 correspond to an alarm device. Furthermore, if the user 10's line of sight is not directed towards the obstacle or the first virtual object, a second virtual object may be generated and displayed to notify the user 10 that an obstacle has been detected in their line of sight. The second virtual object corresponds to obstacle information with an even higher warning level, and includes not only virtual objects displayed in the user's line of sight away from the obstacle, but also virtual objects that are superimposed on or displayed in close proximity to the obstacle, and which are of a different form from the first virtual object.

[0056] In this embodiment, the HMD 100 initiates obstacle notification when it determines that the user 10 is not paying attention to driving. Rather than constantly providing obstacle notifications while driving, by providing obstacle notifications only when the HMD 100 determines that the user 10 is not paying attention to driving, it prevents the user 10 from becoming accustomed to obstacle notifications and becoming less responsive to them, and also enables appropriate driving assistance when the user 10 is not paying attention.

[0057] In this embodiment, the HMD 100 determines whether or not the user 10's attention is decreasing based on the results of the gaze retention determination process (see Figure 3), the results of the body movement determination process (see Figure 4), or a combination of both.

[0058] (Eye-Looking Dwell Time Determination Process) Figure 3 is a schematic diagram illustrating the overview of the eye-looking dwell time determination process.

[0059] Generally, when user 10's concentration on driving is not diminished, as shown in the normal state in Figure 3, they will pay attention not only to the direction of travel but also to the left and right surroundings, and be aware of obstacles such as sudden appearances. However, when drowsiness or fatigue accumulates and attention to driving decreases, the driver's awareness becomes hazy and there is a tendency for eye movement to decrease, as shown in the field of vision stagnation state in Figure 3.

[0060] Therefore, the gaze retention determination unit 1152 of the HMD 100 determines that the gaze is lingering if the direction of the user's gaze detected by the left eye gaze sensor 103 and the right eye gaze sensor 104 remains within a specific narrow field of view area 121 for a predetermined period of time.

[0061] In Graph 202 of Figure 3, the vertical axis represents the field of view angle of user 10, and the horizontal axis represents the measurement time of user 10's gaze direction.

[0062] The gaze dwell determination unit 1152 determines that the gaze is dwelling if the amount of displacement of the field of view angle from the current time t2 to time t1 (which is the time t1 preceding the gaze dwell time threshold Tth) falls within the gaze dwell determination range 201, that is, if the maximum amount of displacement of the field of view angle from time t1 to time t2 is less than the gaze dwell determination range 201. When the gaze dwell determination unit 1152 determines that the gaze is dwelling, the HMD 100 starts obstacle notification processing. Obstacle notification processing in this embodiment means processing to inform the user of the presence of an obstacle by displaying at least one of the first virtual object or the second virtual object, or by emitting a warning sound. The HMD 100 may start obstacle detection processing after determining that the gaze is dwelling, or it may execute obstacle detection processing continuously, but notifications regarding obstacles are executed after the gaze has dwelled.

[0063] In the case of vehicles that maintain balance by shifting their center of gravity, such as motorcycles, they are more susceptible to road surface conditions than four-wheeled vehicles. Therefore, users 10 riding in vehicles that involve shifting their center of gravity tend to look exploratoryly at the road surface below. Consequently, when the user's attention decreases, their gaze tends to linger downwards. Therefore, graph 202 in Figure 3 illustrates the displacement of the field of view angle when the gaze moves vertically. When measuring gaze dwell time, by measuring the gaze dwell time in the vertical gaze direction where the user's gaze tends to linger, it is possible to quickly and accurately detect the decrease in the user's attention and perform field detection of surrounding obstacles without delay.

[0064] As a variation of the above, a three-dimensional region that can be represented by the displacement of the field of view angle in the left-right and up-down-left-right directions may be defined as the gaze dwell determination range, or a gaze dwell determination range that can be represented by the displacement of the field of view angle in the left-right direction may be defined.

[0065] When the gaze dwell determination unit 1152 determines that the user 10's gaze is dwelling, the obstacle detection processing unit 1154 starts detecting obstacles that would obstruct travel in the direction of travel using the camera 101 and the distance measuring sensor 102. When the obstacle detection processing unit 1154 detects an obstacle, the first virtual object processing unit 1155 generates a first virtual object (corresponding to obstacle information) consisting of a frame object surrounding the detected obstacle and an obstacle contour enhancement object that emphasizes the contour of the detected obstacle, and displays it superimposed on the obstacle. Alternatively, the warning notification processing unit 1156 may output an audio warning from the speaker 107 that there is an obstacle in the direction of travel to notify the user, or it may use vibration from the vibrator 108, or flash or light up the lamp 110 to indicate that there is an obstacle in the direction of travel, or it may display a second virtual object. Obstacles may include not only pedestrians and vehicles, but also road surface conditions such as bumps and sinkholes on the road surface that could disrupt the center of gravity balance, such as steps and unevenness in the road surface.

[0066] (Body movement detection process) Figure 4 is a schematic diagram illustrating the overview of the body movement detection process.

[0067] During normal, stable driving, the tilt of the user's body and the vehicle, steering, and weight shifts are repeated periodically, and the tilt of the user's body is also repeated periodically and stably. On the other hand, when riding a vehicle that involves shifting the center of gravity, if the user's attention decreases and it becomes difficult to maintain balance, the user's body will sway irregularly, and the periodicity of the lateral movement and the amplitude of the tilt of the user's body will be disrupted.

[0068] The body movement detection unit 1153 then uses the posture sensor 105 to detect the lateral movement (swaying) of the user 10's body and measures the lateral movement period and tilt amplitude of the user 10's body. When the body movement detection unit 1153 detects irregularities in the lateral movement period and tilt amplitude of the user 10's body, it starts detecting obstacles that would hinder travel in the direction of travel (forward) using the camera 101 and the distance measuring sensor 102.

[0069] Furthermore, the HMD 100 may be configured to use both the body motion detection process and the gaze dwell detection process described in Figure 3 in combination, and to initiate obstacle notification processing when unstable movement and gaze dwell are detected simultaneously, or it may be configured to initiate obstacle notification processing using only the result of the body motion detection process.

[0070] Figure 5 illustrates a state where the user's lateral movement period and tilt amplitude are regular (normal state). Figure 6 illustrates a state where the user's lateral movement period and tilt amplitude are irregular (unstable state).

[0071] In graphs 303 and 404 of Figures 5 and 6, the vertical axis represents the tilt angle of the user 10's body, and the horizontal axis represents the measurement time of the user 10's body tilt angle. In graph 303 of Figure 5, there is no disturbance in the lateral movement period and tilt amplitude of the body. In this case, there is no decrease in the user 10's attention, and it shows the change in the user 10's body tilt angle during normal stable driving conditions. Thus, in Figure 5, the lateral tilt of the user's body is almost constant in both the movement period 301 and tilt amplitude 302, and there is no irregular disturbance in the lateral movement period and tilt amplitude of the body.

[0072] On the other hand, graph 404 in Figure 6 shows irregularities in the lateral movement period and tilt amplitude. Graph 404 shows the change in the tilt angle of user 10's body when user 10 is driving while lacking attention. In this case, the lateral movement period of user 10's body is irregularly disturbed, becoming shorter or longer than the movement period 301 during stable driving, as shown in the area enclosed by the dashed lines 401 and 403. Similarly, the tilt amplitude is irregularly disturbed, becoming larger or smaller than the tilt amplitude 302 during stable driving, as shown in the area enclosed by the dashed lines 401, 402, and 403.

[0073] If the body movement determination unit 1153 determines that the user 10's body is moving unstably, the obstacle detection processing unit 1154 will start detecting obstacles, similar to the result of the gaze dwell determination process. The first virtual object processing unit 1155 will generate and display obstacle information consisting of virtual objects, or the warning notification processing unit 1156 will issue a warning by lighting up, vibrating, or making an audible sound.

[0074] Refer to Figures 7 and 8 to explain examples of how obstacle information is displayed. Figure 7 shows an example of how obstacle information is displayed when the user's gaze is directed towards it. Figure 8 shows an example of how obstacle information is displayed when the user's gaze is not directed towards it.

[0075] Figure 7 shows that, as explained in Figures 1 and 4, when a person 122 suddenly emerges from an alley as an example of an obstacle that obstructs movement in the direction of travel, at least one of the frame object 131 surrounding the person 122 and the obstacle contour enhancement object 132 that emphasizes the contour of the person 122 are superimposed on the actual object, the person 122, as obstacle information and displayed on the HMD 100's display screen 130, and the user's gaze 501 moves and faces the obstacle information or the obstacle itself.

[0076] The warning notification processing unit 1156 obtains the location information of the obstacle from the obstacle detection processing unit 1154, or the display position of the obstacle information from the first virtual object processing unit 1155, and compares these with the gaze direction of the user 10 obtained from the left eye gaze sensor 103 and the right eye gaze sensor 104. If the gaze direction of the user 10 coincides with the location of the obstacle or the display position of the obstacle information, or if the difference between the angle of the gaze direction of the user 10 relative to the HMD 100 and the angle of the location of the obstacle or the display position of the obstacle information relative to the HMD 100 falls within a predetermined allowable angle range, the unit determines that the gaze direction of the user 10 is directed towards the obstacle or obstacle information.

[0077] For example, as shown in Figure 7, when the user's line of sight 501 is directed towards obstacle information or the obstacle itself, it can be presumed that the user 10 is clearly seeing the obstacle in the direction of travel. Therefore, the first virtual object processing unit 1155 superimposes a first virtual object that notifies the person 122, who is the obstacle, of the detection of the obstacle, and performs obstacle notification.

[0078] On the other hand, if the warning notification processing unit 1156 determines that the difference between the angle of the user 10's line of sight relative to the HMD 100 and the angle of the position of the obstacle or the display position of the information obstacle relative to the HMD 100 deviates from a predetermined allowable angle range, it displays a second virtual object containing warning information in the user 10's line of sight, or provides notification in a manner different from the display of the first virtual object, such as emitting a warning, vibrating, or flashing a light.

[0079] For example, as shown in Figure 8, if the user's line of sight 502 is not directed towards obstacle information or an obstacle but towards another line of sight area 503, it can be inferred that the user 10 has not seen an obstacle in the direction of travel. Therefore, the warning notification processing unit 1156 of the HMD 100 notifies the user 10 of the increased warning level by generating and displaying obstacle information (second virtual object) with an increased warning level, emitting a loud sound or voice to indicate the increased warning level, giving the user's body a strong vibration with the vibrator 108, or increasing the brightness of the lamp 110 or flashing it rapidly.

[0080] As shown in Figure 8, the HDM 100 displays, for example, a flashing group display 504 that flashes like lightning surrounding the obstacle, or a star group display 505 that sparkles like stars around the obstacle, as obstacle information for a second virtual object whose warning level has been increased. In addition, as a sound or voice to notify the user of the increased warning level, a loud warning sound like a siren or a warning voice such as "Obstacle ahead!" is emitted. Thus, even if the user 10 does not move and their gaze is directed towards the obstacle information or the obstacle itself, they can easily recognize the presence of an obstacle in the direction of travel through the virtual object obstacle information with an increased warning level, the sound or voice notifying the increased warning level, and the vibration transmitted to their body, and can sense the danger of the obstacle and drive safely. Note that when driving a vehicle that involves shifting the center of gravity, text warnings may be difficult to read, so as obstacle information for a virtual object with an increased warning level, text characters 506 indicating a warning consisting of a virtual object, such as "Watch out for pedestrians suddenly appearing!" may be displayed.

[0081] Next, the operation of the information processing device according to this embodiment will be explained using a flowchart with reference to Figures 9 to 12.

[0082] (Obstacle notification processing based on gaze dwell detection processing) Referring to Figure 9, the flow of obstacle notification processing using the results of gaze dwell detection processing will be explained. Figure 9 is a flowchart showing the flow of obstacle notification processing using both gaze dwell detection processing and body movement detection processing.

[0083] As shown in Figure 9, the HMD 100 waits to process until the user 10 wearing the HMD 100 gets on a vehicle that involves weight shifting (e.g., a motorcycle) and starts moving (S101: No). When the user 10 starts moving (S101: Yes), the calibration processing unit 1151 performs so-called gaze direction calibration, appropriately and optimally setting the gaze dwell determination range 201 and the gaze dwell time threshold Tth according to at least one of the conditions during past driving or the conditions at the start of driving (S102). The left eye gaze sensor 103 and the right eye gaze sensor 104 then detect the gaze direction of the user 10 (S103).

[0084] The gaze dwell time determination unit 1152 acquires the gaze direction information of the detected user 10 and measures the time (gaze dwell time) T during which the angular displacement of the gaze direction remains within the gaze dwell time determination range 201 (S104).

[0085] When the gaze dwell time determination unit 1152 determines that the gaze dwell time T is equal to or greater than the gaze dwell time threshold Tth (S105: Yes), the obstacle detection unit 1154 performs object recognition processing based on the image from the camera 101, detects the distance to the recognized object based on the distance measurement information from the distance measurement sensor 102, and performs so-called obstacle detection processing to determine whether the object is a road obstacle. If there is an obstacle in the direction of travel (S106: Yes), the first virtual object processing unit 1155 generates and displays obstacle information consisting of a first virtual object (S107). Therefore, the camera 101 and the distance measurement sensor 102 correspond to obstacle sensors.

[0086] On the other hand, if the gaze dwell time determination unit 1152 determines that the gaze dwell time T is less than the gaze dwell time threshold Tth (S105: No), and if obstacle information has already been displayed, it stops displaying the obstacle information (S110) and then returns to step S103. For example, if user 10 lacks attention to driving and their gaze is lingering, obstacle information is displayed temporarily. However, when user 10 regains attention and begins to look around in the direction of travel, the gaze dwell time T becomes less than the gaze dwell time threshold Tth, and the displayed obstacle information disappears.

[0087] Furthermore, if the obstacle detection processing unit 1154 does not detect any obstacles in the direction of travel (S106: No), if obstacle information has already been displayed, the display of the obstacle information is stopped (S111), and the process returns to step S103. The case where the obstacle detection processing unit 1154 has not detected any obstacles in the direction of travel also includes the case where obstacle information was initially displayed, but the user 10 avoided the obstacle and overtook it, resulting in the obstacle no longer being in the direction of travel. In this case, the obstacle information that was initially displayed is stopped by step S111 when the obstacle is overtaken.

[0088] If user 10's gaze is not directed towards the obstacle information (S108: No), the process returns to step S103.

[0089] Furthermore, if user 10's gaze is directed towards the obstacle information (S108: Yes), and user 10 detects the obstacle and continues driving while paying attention to the obstacle (S109: No), the process returns to step S103 and proceeds to routine sequence 610 enclosed by the box. Note that if the user detects the obstacle and continues driving while paying attention to the obstacle (S109: No), user 10 has clearly recognized the obstacle, so the display of the obstacle information may be temporarily stopped and the process returns to step S103.

[0090] When user 10 detects an obstacle and stops driving (S109: Yes), the entire sequence ends.

[0091] According to the processing example in Figure 9, the presence or absence of the user 10's gaze direction is used as an indicator of decreased attention to driving. When the HMD 100 determines that the gaze direction is stagnant, it can alert the user 10 to obstacles by superimposing obstacle information on the obstacle or displaying it in close proximity to the obstacle. In step S102, the gaze direction may be calibrated according to the user 10 in light of past driving conditions and the conditions at the start of driving to improve the accuracy of the gaze direction stagnation determination. Furthermore, the gaze direction may be appropriately and optimally calibrated in light of the conditions at regular intervals during driving and driving conditions above a certain speed. In this case, the system may be configured to return to S102 instead of S103 from steps S108, S109, S110, S111. The gaze direction calibration may be performed at any one or any combination of past driving conditions, conditions at the start of driving, conditions at regular intervals during driving, and driving conditions above a certain speed. This is the same in all embodiments.

[0092] (Obstacle notification processing based on gaze dwell detection processing and body movement detection processing) Referring to Figure 10, the flow of obstacle notification processing using the results of gaze dwell detection processing and body movement detection processing will be explained. Figure 10 is a flowchart of the obstacle notification processing flow using both gaze dwell detection processing and body movement detection processing. Since Figure 10 is a process that adds processing using the results of body movement detection processing to the processing flow of Figure 9, only the processing that differs from Figure 9 will be explained.

[0093] In Figure 10, in addition to the operation described in Figure 9, the calibration processing unit 1151 of the HMD 100 performs motion cycle tilt amplitude calibration, which sets criteria for determining disturbances in the left-right motion cycle and tilt amplitude of the body after the vehicle has traveled with a shift in the center of gravity, according to the conditions during past travel, the conditions at the start of travel, or both (S120).

[0094] Furthermore, following step S104, the HMD 100 detects the lateral movement of the user 10's body using the posture sensor 105, and the body movement determination unit 1153 measures the lateral movement period and tilt amplitude of the user 10's body (S121).

[0095] Then, if the gaze dwell time determination unit 1152 determines that the measured gaze dwell time T is greater than or equal to the gaze dwell time threshold Tth (S105: Yes), and the body movement determination unit 1153 determines that there is a disturbance in the movement period and tilt amplitude, meaning that the left-right movement period and tilt amplitude of the user 10's body are irregular (S122: Yes), then the detection of obstacles that would obstruct movement in the direction of travel is started using obstacle sensors (in this embodiment, the camera 101 and the distance measuring sensor 102) (S106).

[0096] On the other hand, if the body movement determination unit 1153 determines that there is no disturbance in the movement period and tilt amplitude of the user 10's body (S122: No), if obstacle information has already been displayed, the display of obstacle information is stopped (S123), and the process returns to step S103, repeating the routine sequence 620 enclosed in the box, which includes gaze detection, measurement of gaze dwell time, determination of gaze dwell time threshold, measurement and disturbance determination of the body's left-right movement period and tilt amplitude, and obstacle detection.

[0097] As shown in Figure 10, by using both gaze dwell detection processing and body movement detection processing to estimate whether the user 10's attention to driving has decreased, the decrease in the user's attention can be detected with greater accuracy, and the user 10 can be effectively alerted to obstacles. Prior to measuring the lateral movement period and tilt amplitude of the body, calibration of the movement period and tilt amplitude may be performed based on the conditions at regular intervals during driving.

[0098] (Obstacle notification processing based on body motion detection processing) The flow of obstacle notification processing using the results of body motion detection processing will be explained with reference to Figure 11. Figure 11 is a flowchart showing the flow of obstacle notification processing using the results of body motion detection processing.

[0099] In Figure 11, the calibration processing unit 1151 performs motion cycle tilt amplitude calibration after the vehicle has traveled with a shift in the center of gravity, setting criteria for determining disturbances in the left-right movement cycle and tilt amplitude of the body according to either the state during past travel, the state at the start of travel, or both (S120).

[0100] Furthermore, the body movement determination unit 1153 detects the lateral movement of the user 10's body using the posture sensor 105 and measures the lateral movement period and tilt amplitude of the user 10's body (S121). If the body movement determination unit 1153 determines that there is a disturbance in the movement period and tilt amplitude, where the lateral movement period and tilt amplitude of the user 10's body are irregular (S122: Yes), the obstacle detection processing unit 1154 starts detecting obstacles that would obstruct travel in the direction of travel using the camera 101 and the distance measuring sensor 102 (S106).

[0101] On the other hand, if the body movement determination unit 1153 determines that there is no disturbance in the movement period and tilt amplitude of the user 10's body (S122: No), if obstacle information has already been displayed, the display of obstacle information is stopped (S123), and the process returns to step S121, repeating the routine sequence 630 enclosed in the box, which involves measuring the left and right movement period and tilt amplitude of the body, determining disturbances, and detecting obstacles.

[0102] When the obstacle detection processing unit 1154 detects that there is an obstacle in the direction of travel (S106: Yes), the HMD 100 generates obstacle information such as a frame object 131 consisting of a first virtual object surrounding the detected obstacle, and an obstacle contour enhancement object 132 consisting of a first virtual object that emphasizes the contour of the detected obstacle, as shown in Figure 9, and displays them superimposed on or close to the obstacle (S107). Then, when the user's gaze moves toward the obstacle information (S108: Yes) and the user 10 detects the obstacle and stops driving (S109: Yes), the entire sequence ends. If the user's gaze does not move toward the obstacle information (S108: No), the process returns to step S121. If the user 10's gaze moves toward the obstacle information (S108: Yes) and the user 10 detects the obstacle and continues driving while paying attention to the obstacle (S109: No), the process returns to step S121 and proceeds to routine sequence 630.

[0103] On the other hand, if the obstacle detection processing unit 1154 does not detect any obstacles in the direction of travel on the HMD 100 (S106: No), and if obstacle information has already been displayed, it stops displaying the obstacle information (S111) and then returns to step S121.

[0104] As shown in Figure 11, by using the body movement determination unit 1153 to estimate whether the user 10's attention to driving has decreased, it is possible to notify the user of an obstacle using phenomena that are characteristic of the user 10 when they lose attention, especially when driving a vehicle that involves shifts in the center of gravity. Prior to measuring the lateral movement period and tilt amplitude of the body, the movement period and tilt amplitude may be calibrated based on the conditions at regular intervals during driving.

[0105] (A mode for increasing the warning level of obstacle notification) Referring to Figure 12, the process by which the warning notification processing unit 1156 executes a warning with a higher warning level when the HMD 100 displays obstacle information but the user 10 does not see the obstacle information will be described. Figure 12 is a flowchart showing the flow of the obstacle notification process with added warning notification processing.

[0106] As shown in Figure 12, in addition to the operation described in Figure 10, if the user 10's gaze does not move toward the obstacle information (S108: No), the warning notification processing unit 1156 issues an obstacle warning notification with an increased warning level (S131). That is, as shown in Figure 8, the user's gaze 502 is not directed toward the obstacle information consisting of the first virtual object or the obstacle itself, but remains directed toward another narrow line of sight area 503, and the user 10 is not clearly seeing the obstacle in the direction of travel. In this case, the warning notification processing unit 1156 issues a warning notification to inform the user 10 of the presence of the obstacle as described in Figure 8, by generating and displaying obstacle information (second virtual object) with an even higher warning level, or by using a different method than displaying the virtual object (for example, vibration, sound notification, etc.).

[0107] This allows the system to notify the user 10 of the presence of an obstacle by issuing a warning notification, even if the first virtual object processing unit 1155 displays the first virtual object superimposed on or close to an obstacle, but the user 10's gaze is not directed towards the obstacle and they do not see it.

[0108] Figure 12 shows an example in which step S131 is added to the process shown in Figure 10, but step S131 may also be added to the process in Figure 9 or Figure 11.

[0109] With the above configuration, the HMD 100 uses a calibration processing unit 1151 to calibrate the gaze direction and motion period tilt amplitude, enabling accurate detection of dwell time in the gaze direction and disturbances in the user's left-right motion period and tilt amplitude. In this state, the gaze dwell time determination unit 1152 measures the dwell time of the user 10's gaze and determines whether the gaze dwell time is above a threshold. At this time, the gaze dwell time threshold may be set shorter in accordance with the travel speed, as the time until collision with an obstacle is shorter when the travel speed is high. If the gaze dwell time exceeds the gaze dwell time threshold, or if the gaze dwell time exceeds the line dwell time threshold and there are disturbances in the motion period and tilt amplitude, the obstacle detection processing unit 1154 detects obstacles that hinder travel in the direction of travel. When an obstacle is detected around the direction of travel, the first virtual object processing unit 1155 generates obstacle information for a first virtual object consisting of a frame object 131 and an obstacle contour enhancement object 132, etc., and displays it superimposed on the obstacle. This allows the system to not issue obstacle notifications when the user 10 is paying sufficient attention while driving the vehicle, and to issue obstacle notifications only when their attention decreases. This prevents the user 10 from becoming accustomed to obstacle notifications and becoming less responsive to them, thus enabling effective obstacle notifications.

[0110] Furthermore, the body movement determination unit 1153 determines whether or not there is a disturbance in the lateral movement period and tilt amplitude of the user 10's body. This allows for appropriate obstacle notification when the user 10, while riding in a vehicle that involves shifting their center of gravity, has difficulty maintaining balance and their attention is reduced.

[0111] Furthermore, if the user's gaze is not directed toward the first virtual object even after the first virtual object is displayed, the warning notification processing unit 1156 generates and displays a second virtual object with an even higher warning level. This ensures that even if the user's gaze is not directed toward the first virtual object or the obstacle itself, the display of the second virtual object with an increased warning level allows the user 10 to reliably recognize the presence of an obstacle in the direction of travel, enabling them to sense the danger posed by the obstacle and drive safely. In particular, vehicles that involve shifts in the center of gravity are prone to losing balance while driving, making it difficult for the driver to change their posture. As a result, even if an obstacle enters the driver's field of view, there is a delay in turning the head towards the obstacle for central visual confirmation. However, by displaying the second virtual object in the user's line of sight, it becomes easier to notify the user of the presence of an obstacle even if the head is not turned toward the obstacle.

[0112] In the above embodiment, an HMD worn on the head was used as an example of the configuration of the information processing device and information processing system according to this embodiment, but it is just one form of HMD, and a glasses-type digital device may also be used. Furthermore, similar effects and benefits can be obtained with a helmet-type or visor-type digital device which is suitable for riding in a vehicle that involves shifts in the center of gravity.

[0113] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0114] Furthermore, the numbers and messages appearing in the text and diagrams are merely examples, and using different ones will not impair the effects of the present invention.

[0115] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as, for example, integrated circuits, general-purpose processors, or application-specific processors. A processor includes transistors and other circuits and is considered a circuit or processing circuit. Alternatively, each of the above configurations and functions may be implemented in software by the processor interpreting and executing programs that realize each function. Information such as programs, tables, and files that realize each function may be stored in memory, recording devices such as hard disks and SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs, or in devices on a communication network. Also, control lines and information lines are shown only if deemed necessary for explanation, and not all control lines and information lines are necessarily shown in the actual product. In practice, almost all configurations can be considered interconnected.

[0116] The above embodiment includes the following form: (Note 1) An information processing device comprising: a processor; a gaze sensor that detects the gaze direction of the user of the information processing device and outputs gaze direction information; an obstacle sensor that detects obstacles around the user's direction of travel and outputs obstacle information; and a display interface, wherein the processor measures the dwell time during which the user's gaze direction remains within a predetermined dwell time determination range based on the gaze direction information obtained from the gaze sensor, and when it is determined that the measured dwell time is equal to or greater than a threshold, it causes the display interface to display a first virtual object superimposed on or near an obstacle detected by the obstacle sensor, based on the obstacle information obtained from the obstacle sensor.

[0117] (Note 2) An information processing device comprising: a processor; a posture sensor that detects the movement of the user of the information processing device and outputs posture information; an obstacle sensor that detects obstacles around the user's direction of travel and outputs obstacle information; and a display interface, wherein the processor determines, based on the posture information obtained from the posture sensor, whether at least one of the period and amplitude of the user's body movement deviates from an operation determination range defined for determining that the user's body movement is stable; and, if it is determined that the user's body movement deviates from the operation determination range, displays a virtual object that includes the obstacle or superimposes on the obstacle on the display interface based on the obstacle information obtained from the obstacle sensor.

[0118] (Note 3) An information processing method, wherein the processor performs the steps of: acquiring user's gaze direction information from a gaze sensor that detects the user's gaze direction; determining, based on the gaze direction information, whether the dwell time during which the angular displacement of the user's gaze direction remains within a predetermined range exceeds a threshold; and, if it is determined that the dwell time exceeds the threshold, displaying a virtual object superimposed on or displayed in close proximity to an obstacle on a display interface, based on obstacle information acquired from an obstacle sensor that detects obstacles around the user's direction of travel.

[0119] (Note 4) An information processing method, wherein the processor performs the steps of: acquiring posture information from a posture sensor that detects the movement of a user's body; determining, based on the posture information, whether at least one of the period and amplitude of the user's body movement has deviated from a predetermined stable operation determination range for determining that the user's body movement is stable; and, if it is determined that the user's body movement has deviated from the stable operation determination range, displaying a virtual object superimposed on or displayed in close proximity to an obstacle on a display interface, based on obstacle information acquired from an obstacle sensor that detects obstacles around the user's direction of travel.

[0120] (Note 5) An information processing system comprising an information terminal equipped with a processor and a wearable display device equipped with an eye-tracking sensor and a display interface, wherein the processor performs the following steps: acquiring user's eye-tracking direction information from the eye-tracking sensor that detects the user's eye-tracking direction; determining, based on the eye-tracking direction information, whether the eye-tracking dwell time, during which the angular displacement of the user's eye-tracking direction remains within a predetermined eye-tracking dwell time determination range for determining whether the eye is dwelling, is equal to or greater than a predetermined eye-tracking dwell time threshold for determining whether the eye is dwelling; and, if it is determined that the eye-tracking dwell time is equal to or greater than the eye-tracking dwell time threshold, causing the display interface to display a virtual object superimposed on or displayed in close proximity to an obstacle, based on obstacle information acquired from an obstacle sensor that detects obstacles around the user's direction of travel.

[0121] (Note 6) An information processing system comprising a communication connection between an information terminal equipped with a processor and a wearable display device equipped with a posture sensor and a display interface, wherein the processor performs the steps of: acquiring posture information from the posture sensor that detects the movement of the user's body; determining, based on the posture information, whether at least one of the period and amplitude of the user's body movement deviates from a predetermined stable operation determination range for determining that the user's body movement is stable; and, if it is determined that the user's body movement has deviated from the stable operation determination range, causing the display interface to display a virtual object superimposed on or displayed in close proximity to the obstacle, based on obstacle information acquired from an obstacle sensor that detects obstacles around the user's direction of travel.

[0122] 1: Information processing system, 10: User, 100: HMD, 101: Camera, 102: Distance sensor, 103: Left eye gaze sensor, 104: Right eye gaze sensor, 105: Attitude sensor, 1051: Acceleration sensor, 1052: Gyroscope sensor, 1053: Geomagnetic sensor, 106: Microphone, 107: Speaker, 108: Vibrator, 109: Positioning sensor, 110: Lamp, 111: Operation input interface, 112: Network communication interface, 113: Short-range wireless communication interface, 114: Display interface, 115: Processor, 1151: Calibration processing unit, 1152: Gaze dwell detection unit, 1153: Body movement detection unit Fixed section, 1154: Obstacle detection processing unit, 1155: First virtual object processing unit, 1156: Warning notification processing unit, 116: Memory, 1162: Operating system, 1163: Application program, 1165: Information data, 1166: Gaze detection information, 1167: Gaze dwell time information, 1168: Motion period tilt amplitude information, 1169: Obstacle information, 117: Timer, 118: Bus, 120: Motorcycle, 121: Field of view area, 122: Person, 130: Display screen, 131: Frame object, 132: Obstacle contour enhancement object, 191: External network, 192: Information server, 193: Information terminal, 201: Gaze dwell detection range, 202: Graph, 301: Motion cycle, 302: Slope amplitude, 303: Graph, 401: Dotted line, 402: Dotted line, 403: Dotted line, 404: Graph, 501: User gaze, 502: User gaze, 503: Gaze area, 504: Flashing group display, 505: Star group display, 506: Text characters, 610: Routine sequence, 620: Routine sequence, 630: Routine sequence

Claims

1. An information processing device comprising: a processor; a gaze sensor that detects the gaze direction of a user of the information processing device and outputs gaze direction information; an obstacle sensor that detects obstacles around the user's direction of travel and outputs obstacle information; and a display interface, wherein the processor measures the dwell time during which the user's gaze direction remains within a predetermined dwell time determination range based on the gaze direction information obtained from the gaze sensor, and when it determines that the measured dwell time is equal to or greater than a threshold, it causes the display interface to display a first virtual object superimposed on or near an obstacle detected by the obstacle sensor, based on the obstacle information obtained from the obstacle sensor.

2. An information processing apparatus according to claim 1, wherein the dwell time determination range is defined to include an angular range in the vertical direction, and the processor measures the time during which the angular displacement of the user's vertical line of sight remains within the dwell time determination range as the dwell time.

3. An information processing apparatus according to claim 1, wherein the processor determines, based on the line of sight information, whether the first virtual object or the obstacle exists in the line of sight of the user, and if the first virtual object or the obstacle does not exist in the line of sight of the user, the information processing apparatus causes the display interface to display a second virtual object for notifying the user that the obstacle has been detected, positioned in the line of sight of the user.

4. An information processing device according to claim 1, further comprising a warning device including one of a warning sound emitter, a vibrator, or a lamp, or any combination thereof, wherein the processor determines, based on the line of sight information, whether the first virtual object or the obstacle exists in the line of sight of the user, and if the first virtual object or the obstacle does not exist in the line of sight of the user, causes the warning device to issue a warning.

5. An information processing device according to claim 1, wherein the processor changes the set value of either the dwell determination range or the threshold, or both, according to the user's past driving state, state at the start of driving, state at regular intervals during driving, or driving state at a certain speed or higher.

6. An information processing device according to claim 1, further comprising a posture sensor that detects the movement of the user's body and outputs posture information, wherein the processor determines that the dwell time has exceeded the threshold and, based on the posture information obtained from the posture sensor, determines that at least one of the period and amplitude of the user's body movement has deviated from an operation determination range defined for determining that the user's body movement is stable, and then displays the first virtual object on the display interface.

7. An information processing device according to claim 6, wherein the processor determines, based on the line of sight information, whether the first virtual object or the obstacle exists in the line of sight of the user, and if the first virtual object or the obstacle does not exist in the line of sight of the user, the information processing device displays a second virtual object on the display interface to notify the user that the obstacle has been detected in the line of sight of the user.

8. An information processing device according to claim 6, further comprising a warning device including one of a warning sound emitter, a vibrator, or a lamp, or any combination thereof, wherein the processor determines, based on the line of sight information, whether the first virtual object or the obstacle is present in the line of sight of the user, and if the first virtual object or the obstacle is not present in the line of sight of the user, causes the warning device to issue a warning.

9. An information processing device according to claim 6, wherein the processor changes the set value of the stable operation determination range according to the user's past driving state, state at the start of driving, state at regular intervals during driving, or driving state at a certain speed or higher.

10. Information processing device comprising: a processor; a posture sensor that detects the movement of a user of the information processing device and outputs posture information; an obstacle sensor that detects obstacles around the user's direction of travel and outputs obstacle information; and a display interface, wherein the processor determines, based on the posture information obtained from the posture sensor, whether at least one of the period and amplitude of the user's body movement deviates from an operation determination range defined for determining that the user's body movement is stable; and, if it determines that the user's body movement deviates from the operation determination range, causes the display interface to display a first virtual object superimposed on or near the obstacle, based on the obstacle information obtained from the obstacle sensor.

11. An information processing method comprising: a processor performing the steps of: acquiring user gaze direction information from a gaze sensor that detects the direction of the user's gaze; determining, based on the gaze direction information, whether the dwell time during which the angular displacement of the user's gaze direction remains within a predetermined range exceeds a threshold; and, if it is determined that the dwell time exceeds the threshold, displaying a virtual object superimposed on or displayed in close proximity to an obstacle on a display interface, based on obstacle information acquired from an obstacle sensor that detects obstacles around the direction of travel of the user.

Images