Warning system, warning method, and program

Abstract

To secure safety for a user who cannot directly view an area that may not be safe.SOLUTION: The warning system includes a region detection unit configured to detect a region of interest that is a region that may be unsafe for a user in a real space, a distance determination unit configured to determine a distance between the user and the region of interest, and a vibration generation unit configured to generate a vibration for causing the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a warning system, a warning method, and a program. [Background technology]

[0002] In recent years, virtual reality (VR) technology has emerged, which displays images of virtual space as if they were real events. Augmented reality (AR) technology also exists, which displays various information superimposed on images of real space. Mixed reality (MR) technology also exists, which displays real space superimposed on virtual space. Head-mounted displays (HMDs) have been developed as devices to realize these technologies. In particular, VR-compatible HMDs block out external visual information from the user, allowing the user to concentrate on viewing or working on content, achieving a high level of immersion.

[0003] However, viewing VR images while blocking external visual information can be dangerous for users. For example, while wearing an HMD, users cannot see the outside world (real space). Therefore, if a user stretches their arms or moves their body while playing a game, they may come into contact with obstacles around them or drop objects on their desk.

[0004] Patent document 1 proposes a device that synthesizes a virtual object at a "position according to the distance to an obstacle" in a virtual space displayed on the HMD to prevent a user wearing the HMD from coming into contact with an obstacle. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-257716 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, obstacles are displayed as virtual objects in a virtual space displayed on an HMD, encouraging the user to avoid the obstacles. However, if the obstacle is not included in the user's field of view (for example, if an obstacle is present behind the user when the user moves backward), it is difficult for the user to notice the obstacle while wearing the HMD.

[0007] Therefore, an object of the present invention is to ensure safety for users who cannot directly view areas that may be unsafe. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. an area detection means for detecting an area of ​​interest in real space that may be unsafe for a user; distance determination means for determining the distance between the user and the region of interest; a vibration generating means for generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; The warning system is characterized by having:

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a region detection step of detecting a region of interest in real space, which is a region that may be unsafe for the user; a distance determination step of determining a distance between the user and the region of interest; a vibration generating step of generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; The warning method is characterized by comprising: [Effects of the Invention]

[0010] According to the present invention, it is possible to ensure safety for users who cannot directly view areas that may be unsafe. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a configuration diagram of a warning system according to a first embodiment. [Figure 2] 1 is a cutaway view of an HMD according to a first embodiment. [Figure 3] 1A to 1C are diagrams illustrating the principle of a gaze detection method according to a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating gaze detection according to the first embodiment. [Figure 5] 4 is a flowchart of a gaze detection operation according to the first embodiment. [Figure 6] 1 is a diagram illustrating a vibration pattern of an illusionary tactile force sense according to embodiment 1. FIG. [Figure 7] 4 is a flowchart of the overall process for obstacle avoidance according to the first embodiment. [Figure 8] 10A and 10B are diagrams illustrating detection of movement using a sensor according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Embodiment 1> A first embodiment will be described in detail with reference to the drawings. In the first embodiment, a warning system having an information processing device will be described. Also, an HMD (head mounted display), which is a head-mounted display device, will be described as an example of the information processing device.

[0013] In the first embodiment, the HMD has a warning unit that warns the user of contact (collision) with an obstacle in a non-transparent mode in which an image of a virtual space can be viewed. Meanwhile, the technology described in the first embodiment is also applicable to a transparent mode related to AR or MR in which an image of a virtual space is projected onto a real space. Here, the "transparent mode" is a mode in which the user can view the real space directly or indirectly. The "non-transparent mode" is a mode in which the user cannot view the real space either directly or indirectly. Below, an HMD that can switch between a transparent mode and a non-transparent mode will be described.

[0014] The configuration of a warning system 1 according to the first embodiment will be described with reference to Fig. 1. The warning system 1 includes an HMD 100, a controller 113a, and a controller 113b. Fig. 1 shows the configuration of the HMD 100 as seen from the top of the user's head (Y-axis direction).

[0015] (HMD configuration) The HMD 100 has a housing 103, a left-eye display 104, a right-eye display 105, a left-eye camera 106, a right-eye camera 107, a left-eye gaze detector 108, a right-eye gaze detector 109, an inertial sensor 110, a tactile illusion unit 111, and a communication unit 112.

[0016] The housing 103 fixes (holds) each component of the HMD 100.

[0017] The left-eye display 104 displays an image for viewing by the user's left eye 101. If the MD 100 is in the see-through mode, when the housing 103 of the HMD 100 is worn on the head, the left eye 101 can observe the real space through the left eye display 104 .

[0018] The right-eye display 105 displays an image to be viewed by the user's right eye 102. If the HMD 100 is in the see-through mode, when the housing 103 of the HMD 100 is worn on the head, the right eye 102 can observe the real space through the right-eye display 105. In the see-through mode, when images such as operation icons are displayed on each display, the user can view the real space and the operation icons through the displays.

[0019] The left-eye camera 106 captures an image of the real space. In the transmission mode, the image of the real space captured by the left-eye camera 106 is displayed on the left-eye display 104.

[0020] Right-eye camera 107 captures an image of real space. In the transmission mode, the image of real space captured by right-eye camera 107 is displayed on right-eye display 105.

[0021] The left-eye gaze detector 108 can detect the gaze direction of the user's left eyeball 101 and the position (viewpoint position) at which the user's left eyeball 101 is looking. The right-eye gaze detector 109 can detect the gaze direction of the user's right eyeball 102 and the position (viewpoint position) at which the user's right eyeball 102 is looking.

[0022] The inertial sensor 110 has an acceleration sensor that detects translational motion of the HMD 100 on each of the X, Y, and Z axes. Furthermore, the inertial sensor 110 has a gyro sensor that detects rotational motion of the HMD 100 on each of the YPR (yaw, pitch, roll) axes. By associating these two sensors, the inertial sensor 110 can comprehensively detect the translational motion and rotational motion of the HMD 100 (= the translational motion and rotational motion of the user's head).

[0023] The illusionary tactile force sense unit (vibration generating unit) 111 is a haptic device that realizes haptic technology (illusory tactile force sense technology). The illusionary tactile force sense unit 111 includes an acceleration sensor, a position sensor, and an eccentric motor. The illusionary tactile force sense unit 111 acquires acceleration information and position information of the body part in contact, and generates various vibrations using the eccentric motor according to the acquired information. In this way, the illusionary tactile force sense unit 111 makes the user feel the sensation of being pushed or pulled.

[0024] The illusionary tactile force sense unit 111 can provide the user with perceptual warnings (instructions) regarding obstacle avoidance by using vibration patterns that utilize the haptic effect. The illusionary tactile force sense unit 111 may not only be a circuit provided in the HMD 100, but may also be mounted (equipped) on the controller 113a or the controller 113b. Furthermore, multiple motion sensors equipped with the illusionary tactile force sense unit 111 may be independently worn (equipped) near each joint of the user's body. In this way, the motion sensors can estimate the user's movements, including their posture, by performing three-dimensional posture estimation, and generate fine illusionary tactile force sense vibrations for each wearing position. This allows the illusionary tactile force sense unit 111 to generate pulling tension in a more appropriate direction.

[0025] The communication unit 112 transmits and receives information to and from an external communication device. The HMD 100 can exchange information with devices paired with the HMD 100 (such as the controllers 113a and 113b) via the communication unit 112.

[0026] (Controller configuration) The controller 113a and the controller 113b are controllers for controlling game images or images in a virtual space. The controller 113a is held in the user's right hand. Controller 113a is a controller that is held in the user's left hand. Controller 113b is a controller that is held in the user's left hand. Controller 113a and controller 113b have similar configurations, so only the configuration of controller 113a will be described below.

[0027] Controller 113a has a plurality of operation members such as a cross key 114a, buttons 115a, a lever 116a, and a touch panel 117a. By operating each operation member, the user can control, for example, images displayed on left-eye display 104 and right-eye display 105. Operations on controller 113a are transmitted to CPU 128 (see FIG. 2) as operation signals.

[0028] Fig. 2 is a cross-sectional view of the HMD 100 cut into left and right halves along the YZ plane formed by the Y axis and Z axis shown in Fig. 1. Fig. 2 shows a schematic diagram of the mechanism that performs gaze detection. Note that Fig. 2 is a cross-sectional view seen from the left eye side, and the mechanism on the left eye side will be explained below. Note that the mechanism on the right eye side is the same as the mechanism on the left eye side.

[0029] The HMD 100 has a left-eye display 104, an illumination light source 120, a light splitter 121, a light-receiving lens 122, and an eye image sensor 123. The HMD 100 also has a display drive circuit 124, a camera image sensor 125, an aperture mechanism 126, a focus mechanism 127, a CPU 128, and a memory unit 129. The CPU 128 and the memory unit 129 are common to at least the left-eye mechanism and the right-eye mechanism.

[0030] The illumination light source 120 is a light source that projects light onto the left eyeball 101 for gaze detection. The illumination light source 120 includes, for example, a plurality of infrared light emitting diodes.

[0031] The light splitter 121 splits the light from the real space into reflected light and transmitted light.

[0032] The light receiving lens 122 forms an image of the illuminated eyeball and an image resulting from the corneal reflection of the light source onto the ocular imaging element 123. The light receiving lens 122 positions the pupil of the user's left eyeball 101 and the ocular imaging element 123 in a complementary imaging relationship.

[0033] An array of photoelectric elements such as CMOS is arranged two-dimensionally in eye imaging element 123. Based on the positional relationship between the image of the eyeball formed on eye imaging element 123 and the image of illumination light source 120 due to corneal reflection, the gaze direction can be detected using a predetermined algorithm described later.

[0034] The illumination light source 120, the light receiving lens 122, and the eye image sensor 123 constitute a left-eye line-of-sight detector 108.

[0035] Camera imaging element 125, aperture mechanism 126, and focus mechanism 127 are mechanisms that make up left-eye camera 106, which captures an image of the outside (real space) in the transmission mode. Left-eye camera 106 can capture an image of a subject through optical splitter 121.

[0036] The CPU 128 controls the entire HMD 100 .

[0037] The memory unit 129 stores the imaging signals from the camera imaging element 125 and the eye imaging element 123. The memory unit 129 stores the line of sight correction data.

[0038] (Description of gaze detection operation) The gaze detection method will be described with reference to Figs. 3, 4A, 4B, and 5. Fig. 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in Fig. 3, illumination light source 120 (light sources 120a and 120b) is incident on light receiving lens 1. 22 and illuminates the user's eyeball 140. A portion of the light emitted from light sources 120a and 120b and reflected by eyeball 140 is collected by light receiving lens 122 onto ocular imaging element 123. FIG. 4A is a schematic diagram of the eye image captured by ocular imaging element 123 (eyeball image projected onto ocular imaging element 123), and FIG. 4B is a diagram showing the output intensity of the CCD in ocular imaging element 123. FIG. 5 shows a schematic flowchart of the gaze detection operation.

[0039] 5, light sources 120a and 120b emit infrared light toward user's eyeball 140. An image of the user's eyeball illuminated by the infrared light is formed on eye image sensor 123 through light receiving lens 122 and is photoelectrically converted by eye image sensor 123. As a result, an electrical signal of the eye image that can be processed is obtained.

[0040] In step S502, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 123 to the CPU 128.

[0041] In step S503, the CPU 128 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 120a and 120b and the pupil center c from the eye image obtained in step S502.

[0042] Infrared light emitted from light sources 120a and 120b illuminates cornea 142 of user's eyeball 140. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 122 and formed on ocular imaging element 123 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 123 as pupil edge images a' and b' in the eye image.

[0043] FIG. 4B shows luminance information (luminance distribution) of region α' in the eye image of FIG. 4A. In FIG. 4B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd' and Pe' are set to Xd and Xe, and the X-axis coordinates of the pupil edge images a' and b' are set to Xa and Xb. As shown in FIG. 4B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 123), an extremely low level of luminance is obtained except for the coordinates Xd and Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa and the region where the X coordinate is larger than coordinate Xb.

[0044] From the luminance distribution shown in FIG. 4B, the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 122 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 123 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0045] In step S504, CPU 128 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 140 relative to light receiving lens 122, and is the magnification of corneal reflection image Pd' , Pe' can be calculated using a function of the interval (Xd-Xe).

[0046] In step S505, CPU 128 calculates the rotation angle of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 122. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 are approximately the same. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, the rotation angle θ of eyeball 140 in the ZX plane (plane perpendicular to the Y axis) is X can be calculated using the following equation 1. The rotation angle θy of the eyeball 140 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating the rotation angle θx. β×Oc×SINθ X ≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0047] In step S506, the CPU 128 uses the calculated rotation angles θx and θy to determine (estimate) the user's viewpoint position (the position where the line of sight is fixed; the position where the user is looking) in the viewing image displayed on the display unit (such as the left-eye display 104). If the coordinates (Hx, Hy) of the viewpoint position are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint position can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0048] The parameter m in equations 2 and 3 is a constant determined by the configuration of the finder optical system (such as the light receiving lens 122), and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c in the visual image. The parameter m is determined in advance and stored in the memory unit 129. The parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in the gaze, and are acquired by performing a calibration operation and stored in the memory unit 129 before the gaze detection operation starts.

[0049] In step S507, the CPU 128 stores the coordinates (Hx, Hy) of the viewpoint position in the memory unit 129, and ends the gaze detection operation. Note that the gaze direction is the direction from the user's eyeball to the viewpoint position, and therefore can be calculated based on the coordinates of the user's eyeball and the coordinates of the viewpoint position.

[0050] In the above description, the method of acquiring the coordinates of the viewpoint position (point of gaze) on the display unit using the corneal reflection images of the light sources 120a and 120b has been described. However, the present invention is not limited to this, and any method of acquiring the coordinates of the viewpoint position (eyeball rotation angle) from a captured eyeball image may be used.

[0051] (Overall processing explanation) The overall processing of the warning system 1 for obstacle avoidance according to the first embodiment will be described with reference to the flowchart of Fig. 7. The following processing is realized by the CPU 128 controlling each unit in accordance with a program stored in the memory unit 129. The processing of the flowchart of Fig. 7 starts in a state in which "the HMD 100 is powered on, the CPU 128 performs image processing, and an image of a virtual space or the like is displayed on the left-eye display 104 and the right-eye display 105."

[0052] In step S701, the CPU 128 controls the left-eye gaze detector 108 and the right-eye gaze detector 109 to acquire information on the user's gaze direction (gaze information).

[0053] In step S702, the CPU 128 determines whether the user is looking at the screen based on the information about the user's line of sight. The CPU 128 calculates the outside of the visual field range of the user. The "user's visual field range" is, for example, the range of the "user's central visual field" when the user is not wearing the HMD 100. Note that the "user's visual field range" may also be "a range including the user's central visual field and peripheral visual field" when the user is not wearing the HMD 100. Since the central visual field of a human varies from person to person, it is important to calculate the range of the central visual field of the user using the HMD 100 in advance in order to accurately calculate the outside of the visual field range. Furthermore, the CPU 128 may set, for example, the range extending within a predetermined angle range (for example, within 35 degrees) with the user's line of sight (for example, line of sight direction 611 in FIG. 6A) as the center (reference) as the central visual field (for example, the visual field range 612 in FIG. 6A). Then, the CPU 128 may calculate, for example, an area outside the central visual field as the outside of the visual field range.

[0054] In step S703, the CPU 128 detects an "area including an obstacle that requires obstacle avoidance" as a "region of interest," and then calculates (detects) the positional relationship between the region of interest and the user (the distance between the region of interest and the user). For example, an obstacle may be another player (person) who is experiencing the virtual space (virtual reality space) at the same time as the user and who is within the guardian (the range within which the user can move) set by the user. An obstacle may also be a moving object such as a ball that has entered the guardian. In other words, an obstacle may be any object that the user may come into contact with (for example, an object within the guardian, or an object within a specific distance, such as within 1 meter or 2 meters, from the user). In addition to such objects (moving objects), the region of interest may also include, for example, an "object that is difficult to remove before experiencing the virtual space and that is unavoidably present within the guardian (such as a large desk or table set in advance)." The region of interest may also include an "unmovable area outside the guardian." Therefore, the area of ​​interest is not limited to an area where there is a possibility of contact with an obstacle, but may be any area that may be unsafe (dangerous) (for example, an area with a slippery floor or an area with a step). In the following description, the area of ​​one obstacle is assumed to be the area of ​​interest. Furthermore, the distance between the user and the area of ​​interest may be the distance between the user's head (= HMD100) and the center of the area of ​​interest, or the distance between the user's hand or foot and the outer periphery of the area of ​​interest.

[0055] In step S704, CPU 128 determines whether the distance between the user and the region of interest calculated in step S703 is within a predetermined range (less than a predetermined threshold). If it is determined that the distance between the user and the region of interest is far and outside the predetermined range (equal to or greater than the predetermined threshold), it is determined that obstacle avoidance is not necessary, and the process returns to step S701. If it is determined that the distance between the user and the region of interest is within the predetermined range (less than the predetermined threshold), the process proceeds to step S705.

[0056] In step S705, the CPU 128 determines whether the region of interest is located outside (included in) the field of view of the user wearing the HMD 100. If it is determined that the region of interest is located inside (included in) the field of view of the user, the process proceeds to step S706. If it is determined that the region of interest is located outside the field of view of the user, the process proceeds to step S707.

[0057] In step S706, the CPU 128 issues a preset warning (notification of the risk of contact with an obstacle) to urge the user to avoid the obstacle. For example, the CPU 128 displays a virtual object representing an obstacle in an image of the virtual space displayed on the HMD 100. In the non-transparent mode, the user wearing the HMD 100 cannot directly see the actual obstacle. However, if a virtual object is projected (placed) at a position in the virtual space corresponding to the position of the actual obstacle (area of ​​interest), the user can indirectly see the obstacle. This allows the user to avoid the obstacle. Furthermore, since the actual obstacle is not displayed while the virtual space is being displayed, the immersive feeling of the virtual space experience (virtual reality experience) is reduced. It is also possible to prevent the decline.

[0058] In step S707, CPU 128 controls illusionary tactile force sense unit 111 to generate a vibration pattern (a vibration pattern that causes the user to perceive an illusionary tactile force sense) based on the positional relationship between the user and the area of ​​interest. In embodiment 1, illusionary tactile force sense unit 111 generates vibrations based on the "direction in which the area of ​​interest exists relative to the user's position" and the "distance from the user to the area of ​​interest." This causes CPU 128 to issue a warning indicating that there is a risk of contact between the user and an obstacle.

[0059] Specifically, in step S707, the illusionary force sense unit 111 generates a vibration pattern and vibrates with the generated vibration pattern. In this way, the illusionary force sense unit 111 notifies the user of the position of an obstacle in real space. The vibration pattern generated by the illusionary force sense unit 111 is a vibration pattern that can give the user a sense of tactile response or a sensation of touching an object by stimulating the user's skin with a special vibration pattern.

[0060] In the first embodiment, the illusionary force sense unit 111 is mounted (equipped) on the HMD 100. The movement of the head of the user wearing the HMD 100 is sensed by a position sensor or an acceleration sensor. The illusionary force sense unit 111 changes the acceleration pattern of the eccentric motor according to the position, speed, or acceleration information obtained by the sensor. In this way, the illusionary force sense unit 111 can give the user an illusionary force sense. By changing the vibration pattern, the illusionary force sense unit 111 can express a "force sense" such as being pulled or pushed, a "pressure sense" such as soft / hard, and a "tactile sense" which is the feel of the surface material of an object.

[0061] In the first embodiment, the illusionary tactile force sense unit 111 makes the user feel a pulling or pushing force (a force having components of direction and strength) depending on the positional relationship between the user and an obstacle. This allows a user wearing the HMD 100 who cannot see an actual obstacle to intuitively identify the position of the obstacle.

[0062] In this way, the CPU 128 operates as an "area detection unit that detects a region of interest in real space that may be unsafe for the user (such as a region that includes an obstacle that the user may come into contact with)." The CPU 128 also operates as a "determination unit that determines whether the region of interest is included in the user's field of view." The CPU 128 also operates as a "distance determination unit that determines the distance between the user and the region of interest." However, the warning system 1 may have a region detection unit, a determination unit, and a distance determination unit in addition to the CPU 128.

[0063] Hereinafter, with reference to FIGS. 6A to 6C, differences in the vibration patterns of the tactile illusion caused by the positional relationship between the user and an obstacle will be described. FIGS. 6A, 6B, and 6C each show the vibration patterns of the tactile illusion caused by different positional relationships between the user 600 wearing the HMD 100 and the obstacle. A gaze direction 611 indicates the gaze direction of the user 600 wearing the HMD 100, and a visual field range 612 indicates the range of the central visual field that the user 600 can see when not wearing the HMD 100. Furthermore, vectors 621 and 622 represent the magnitude and direction of the pulling (or pushing) force of the tactile illusion caused by the vibration perceived by the user 600. For vectors 621 and 622, a larger norm indicates that a stronger pulling force is perceived as the vibration being generated. Furthermore, ranges 631 and 632 indicate ranges corresponding to the distance from the user 600 to the obstacle.

[0064] FIG. 6A shows a state in which the user 600 moves backward and approaches an obstacle 602 outside the field of view 612. The obstacle 602 is determined to be located outside the field of view 612 of the user 600. In this case, a background motion that moves the user 600 away from the obstacle 602 (region of interest) is performed. The force of the illusionary tactile sensation is generated in a vibration pattern that makes the user 600 perceive a pulling force in the direction indicated by the vector 621. The strength (intensity) of the pulling vibration at this time is controlled so that the closer the user 600 is to the obstacle 602 (focused area), the stronger the force felt.

[0065] Furthermore, FIG. 6B shows a state in which an obstacle 603 is located in the range 631. In this case, the distance between the user 600 and the obstacle 603 is shorter than the distance between the user 600 and the obstacle 602 shown in FIG. 6A, and therefore the user 600 can see that danger is approaching. Therefore, even if the pulling direction that moves the user 600 away from the obstacle 603 (the area of ​​interest) is the same, a haptic vibration that causes the user 600 to feel a stronger pulling force is generated. Furthermore, the illusionary tactile force sense unit 111 may detect a change in the user 600's posture and control the strength of the vibration (the strength of the force caused by the vibration) based on the speed of the user 600's movement. For example, the faster the user 600 moves, the greater the degree of danger to the user 600 when they come into contact with an obstacle. Therefore, the illusionary tactile force sense unit 111 may control the vibration so that the stronger the pulling force applied to the user 600 is the faster the user 600 moves.

[0066] On the other hand, FIG. 6C shows a state in which the user 600 moves forward and approaches an obstacle 601 within the field of view 612. The obstacle 601 is present within the field of view 612 of the user 600. Therefore, in this case, the warning system 1 does not necessarily need to generate haptic vibrations to warn the user; a visual warning is also possible. Therefore, the warning system 1 can issue other types of warnings set by the user 600 in advance. For example, the warning system 1 projects a virtual object representing the obstacle 601 to issue a warning that does not reduce the user 600's sense of immersion in the image of the virtual space being viewed. In this way, the warning system 1 can issue various warnings regarding obstacles within the field of view 612 of the user 600.

[0067] Furthermore, since the sense of distance corresponding to the strength of pulling varies from person to person, calibration may be performed in advance to determine the pulling strength and direction corresponding to a position in space.

[0068] Furthermore, because haptic vibration allows the user to sense the direction and strength of the pull, a warning using a haptic vibration pattern may be issued not only when the area of ​​interest is located outside the user's field of view, but also when the area of ​​interest is located within the field of view. In particular, if the user is not concentrating on an image within the field of view, it may be difficult for the user to notice the risk of contact even if a virtual object representing an obstacle is displayed. Even in such cases, a warning using a haptic vibration pattern allows even a user with distracted attention to grasp the risk of contact.

[0069] The above has described the overall processing of the warning system 1 for obstacle avoidance according to embodiment 1. According to embodiment 1, it is possible to realize a warning system 1 that can issue an appropriate warning about an obstacle located outside the field of view that cannot be visually recognized by the user wearing the HMD 100, while preventing a decrease in the sense of immersion due to a warning about obstacle avoidance. This makes it possible to ensure the safety of movement for a user who cannot directly visually recognize a potentially unsafe area.

[0070] <Modification> Although the first embodiment has been described as an example in which there is one obstacle, the technology according to the first embodiment can also be applied to cases in which there are multiple obstacles in real space. When multiple obstacles exist, the warning system 1 may calculate (determine) the positional relationship between each of the multiple obstacles and the user, and determine the area including the obstacle closest to the user among the multiple obstacles (the area corresponding to that obstacle) as the area of ​​interest. Then, the warning system 1 may control (change) the vibration pattern of the tactile illusion according to the positional relationship between the area of ​​interest and the user. The warning system 1 can warn the user of the risk of contact with the nearest obstacle by using haptic vibration, allowing the user to avoid contact.

[0071] Furthermore, the user's visual field range includes a first range that is easy for the user to see (for example, the user's central visual field) and a second range that is harder to see than the first range (for example, the user's peripheral visual field). Therefore, the warning system 1 (the tactile illusion unit 111) may change the vibration pattern (the strength of the vibration) when the area of ​​interest is located in the first range and when it is located in the second range. For example, when the area of ​​interest is located in the first range, the tactile illusion unit 111 weakens the vibration compared to when the area of ​​interest is located in the second range, because there is a high possibility that the user will be able to recognize contact with an obstacle by the displayed object.

[0072] Furthermore, when generating tactile illusionary force vibrations even when the area of ​​interest is located within the user's field of view, the warning system 1 may change the vibration pattern (such as the intensity of the vibration) of the tactile illusionary force between when the area of ​​interest is located within the user's field of view and when it is not. For example, the warning system 1 weakens the intensity of the tactile illusionary force vibration when the area of ​​interest is located within the user's field of view compared to when the area of ​​interest is located outside the user's field of view. By using such a tactile illusionary force vibration pattern, even when the area of ​​interest is located within the user's field of view and the user is likely to miss the virtual object, the tactile illusionary force vibration can alert the user to the risk of contact. Furthermore, weakening the intensity of the tactile illusionary force vibration also helps prevent a decrease in the sense of immersion in the image in the virtual space. In this way, by using tactile illusionary force vibrations for warnings, the warning system 1 can simultaneously issue warnings using multiple methods, rather than just one.

[0073] In the first embodiment, the warning system 1 includes an HMD 100 equipped with a sensor for detecting user movement and a sensor for generating haptic illusionary vibrations. Furthermore, if the accuracy of user movement detection can be further improved, it is possible to more accurately identify the user's avoidance direction from an obstacle. For example, in addition to the HMD 100, a sensor (acceleration sensor, gyro sensor, depth sensor, GPS) for detecting movement information and a wearable device for generating haptic illusionary vibrations may be attached to the user's right or left wrist or other part of the body. Furthermore, a motion sensor mounted on the controller may be used to estimate the user's three-dimensional posture. This configuration enables warnings tailored to the user's detailed movements. In particular, the wearable device can be freely attached to a part of the body that does not interfere with the user's movements (such as a part near a joint of the human body).

[0074] There are various methods for identifying the part of the body where a wearable device is attached. Methods for identifying the part where a wearable device is attached include, for example, identifying the attachment position based on image data captured by a camera of the wearable device attached to the user, and setting the part where the wearable device is attached in advance. Another method for identifying the part where a wearable device is attached is to identify the part by comparing "pre-recorded magnitude of acceleration or speed of each part of the user's body" with "movement of the wearable device."

[0075] The sensors attached to the HMD 100 can only detect simple movements such as translation, as shown in FIG. 8A. However, if the movement of each part of the user can be detected using an acceleration sensor or gyro sensor, it is possible to detect the acceleration of each part of the user's body when wearing multiple wearable devices. This makes it possible to detect not only translation, but also local movements of the user's hands and movements such as body rotation, as shown in FIG. 8B. As a result, the warning system 1 can generate haptic vibrations that allow the user to perceive a more appropriate direction to avoid contact between the user and an obstacle. For this reason, the warning system 1 can pull the user's body in the appropriate direction for each part, encouraging the user to move in a safe direction. You can guide the

[0076] Although the above description describes a case in which a user viewing an image in a virtual space is notified of the risk of contact with an obstacle using tactile force vibrations, tactile force vibrations may also be used in other cases. For example, when a user is wearing glasses and an obstacle outside the user's field of view approaches the user, the warning system may notify the user of the risk of contact with the obstacle using tactile force vibrations of the glasses. Furthermore, when a visually impaired person is wearing a specific device (haptic device) and an obstacle approaches the user, the warning system may notify the user of the risk of contact with the obstacle using tactile force vibrations of the specific device. Note that because the field of view of a visually impaired person is limited to zero or a small range, a notification using tactile force vibrations may be used regardless of whether the area of ​​interest (obstacle) is located within the field of view, as long as the distance between the user and the area of ​​interest is shorter than a predetermined distance.

[0077] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0078] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0079] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0080] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0081] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit that realizes one or more functions.

[0082] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) an area detection means for detecting an area of ​​interest in real space that may be unsafe for a user; distance determination means for determining the distance between the user and the region of interest; a vibration generating means for generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; 1. A warning system comprising: (Configuration 2) When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means generates vibrations based on a direction in which the region of interest exists relative to the user and a distance from the user to the region of interest. 2. The warning system according to claim 1, (Configuration 3) the area detection means detects, as the area of ​​interest, an area including at least one of an area outside the specific area set by the user, a predetermined object in the specific area, a person other than the user in the specific area, and an object moving in the specific area; 3. The warning system according to configuration 1 or 2, (Configuration 4) When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means detects a change in the user's posture and controls the strength of the vibrations based on the speed of the user's movement. 4. The warning system according to any one of configurations 1 to 3. (Configuration 5) When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means generates stronger vibrations as the distance between the user and the region of interest becomes shorter. 5. A warning system according to any one of configurations 1 to 4. (Configuration 6) the area detection means determines a positional relationship between each of a plurality of obstacles in real space and the user, and detects an area corresponding to an obstacle that is closest to the user as the area of ​​interest; 6. A warning system according to any one of configurations 1 to 5. (Configuration 7) The method further includes a determination unit for determining whether the region of interest is included in the user's visual field range, the vibration generating means generates a vibration that causes the user to perceive a force in a direction away from the region of interest in a first case in which it is determined that the region of interest is not included in the field of view of the user and that the distance between the user and the region of interest is less than the threshold value; 7. A warning system according to any one of configurations 1 to 6. (Configuration 8) the user wears a display device on his / her head, The visual field range is the range of the central visual field when the user is not wearing the display device on their head. 8. The warning system of claim 7. (Configuration 9) The warning system of configuration 7 or 8, characterized in that in a second case where it is determined that the area of ​​interest is included in the user's field of view and the distance between the user and the area of ​​interest is less than the threshold, the vibration generating means does not generate a vibration that causes the user to perceive a force in a direction away from the area of ​​interest. (Configuration 10) the vibration generating means generates a vibration that causes the user to perceive a force in a direction away from the region of interest, even in a second case in which it is determined that the region of interest is included in the field of view of the user and the distance between the user and the region of interest is less than the threshold value; 9. The warning system according to configuration 7 or 8. (Configuration 11) The vibration generating means changes the intensity of the vibration to be generated between the first case and the second case. 11. The warning system of claim 10. (Configuration 12) the field of view range includes a first region and a second region that is less visible to the user than the first region, In the second case, the vibration generating means changes the intensity of the vibration to be generated between when the target area is located in the first area and when the target area is located in the second area. 12. The warning system according to claim 10 or 11. (Configuration 13) further comprising a display means for displaying an image of the virtual space; In the second case, the display means displays an image in which an object is placed at a position in the virtual space corresponding to the position of the region of interest. 13. A warning system according to any one of configurations 9 to 12. (Configuration 14) a head-mounted display and a controller; the vibration generating means is provided on at least one of the head-mounted display, the controller, and the user; 14. A warning system according to any one of configurations 1 to 13. (method) a region detection step of detecting a region of interest in real space, which is a region that may be unsafe for the user; a distance determination step of determining a distance between the user and the region of interest; a vibration generating step of generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; A warning method comprising: (program) A program for causing a computer to function as each means of the warning system according to any one of configurations 1 to 14. [Explanation of symbols]

[0083] 1: Warning system, 111: Illusionary tactile force sensory unit (vibration generating unit), 128:CPU

Claims

1. an area detection means for detecting an area of ​​interest in real space that may be unsafe for a user; distance determination means for determining the distance between the user and the region of interest; a vibration generating means for generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; 1. A warning system comprising:

2. When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means generates vibrations based on a direction in which the region of interest exists relative to the user and a distance from the user to the region of interest.

2. The warning system of claim 1.

3. the area detection means detects, as the area of ​​interest, an area including at least one of an area outside the specific area set by the user, a predetermined object in the specific area, a person other than the user in the specific area, and an object moving in the specific area; 2. The warning system of claim 1.

4. When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means detects a change in the user's posture and controls the strength of the vibrations based on the speed of the user's movement.

2. The warning system of claim 1.

5. When generating vibrations that cause the user to perceive a force in a direction away from the region of interest, the vibration generating means generates stronger vibrations as the distance between the user and the region of interest becomes shorter.

2. The warning system of claim 1.

6. the area detection means determines a positional relationship between each of a plurality of obstacles in real space and the user, and detects an area corresponding to an obstacle that is closest to the user as the area of ​​interest; 2. The warning system of claim 1.

7. The method further includes a determination unit for determining whether the region of interest is included in the user's visual field range, the vibration generating means generates a vibration that causes the user to perceive a force in a direction away from the region of interest in a first case in which it is determined that the region of interest is not included in the field of view of the user and that the distance between the user and the region of interest is less than the threshold value; 2. The warning system of claim 1.

8. the user wears a display device on his / her head, The visual field range is the range of the central visual field when the user is not wearing the display device on their head.

8. The warning system of claim 7.

9. The vibration generating means is configured to generate vibrations such that the area of ​​interest is included in the visual field of the user, and The warning system of claim 7, characterized in that in a second case where it is determined that the distance between the user and the area of ​​interest is less than the threshold, no vibration is generated that causes the user to perceive a force in a direction away from the area of ​​interest.

10. the vibration generating means generates a vibration that causes the user to perceive a force in a direction away from the region of interest, even in a second case in which it is determined that the region of interest is included in the field of view of the user and the distance between the user and the region of interest is less than the threshold value; 8. The warning system of claim 7.

11. the vibration generating means changes the intensity of the vibration to be generated between the first case and the second case.

11. The warning system of claim 10.

12. the field of view range includes a first region and a second region that is less visible to the user than the first region, In the second case, the vibration generating means changes the intensity of the vibration to be generated between when the target area is located in the first area and when the target area is located in the second area.

11. The warning system of claim 10.

13. further comprising a display means for displaying an image of the virtual space; In the second case, the display means displays an image in which an object is placed at a position in the virtual space corresponding to the position of the region of interest.

10. The warning system of claim 9.

14. a head-mounted display and a controller; the vibration generating means is provided on at least one of the head-mounted display, the controller, and the user; 2. The warning system of claim 1.

15. a region detection step of detecting a region of interest in real space, which is a region that may be unsafe for the user; a distance determination step of determining a distance between the user and the region of interest; a vibration generating step of generating a vibration that causes the user to perceive a force in a direction away from the region of interest based on a positional relationship between the user and the region of interest when it is determined that the distance between the user and the region of interest is less than a threshold; A warning method comprising:

16. A program for causing a computer to function as each means of the warning system according to any one of claims 1 to 14.

Images