Wearable terminal and information processing method

Abstract

Provided is a wearable terminal worn by a user, said wearable terminal comprising: a display which displays, on a display screen, a virtual space which may include a virtual object, an image of a real space and a virtual object, or only an image of a real space; a wired connection interface to which a wired cable is connected and which is used for a wired connection with a device; and a processor. When the processor identifies that the wired connection with the device has been established and identifies that the position of the device is at least a prescribed distance away, the processor provides the user with a notification encouraging the user to pay attention to the use of the wearable terminal.

Description

Wearable device and information processing method

[0001] The present invention relates to a wearable terminal and an information processing method.

[0002] There is known a technology for experiencing virtual reality (VR) by generating a virtual space and displaying it on a display. There is also known an augmented reality (AR) technology for superimposing virtual objects created using CG (Computer Graphics) or the like on an image of real space. These technologies are widely used, and wearable devices worn by users are becoming widespread. One example of such a device is a head-mounted display (HMD) worn by a user on the head. HMDs can be used, for example, for gaming experiences, simulations, and the like.

[0003] Patent document 1 discloses a technology related to security when transmitting power wirelessly, and discloses a technology in which, if a smartwatch determines that it is a predetermined distance away from a charging device, it determines that a third party has taken the charging device and issues a warning.

[0004] JP 2016-110342 A

[0005] A wearable device may be connected to various devices such as a controller, a charger, and earphones via a wired cable. As an example, while the wearable device is being used (for example, while the user is moving around using the wearable device), the wired cable may be pulled unintentionally by the user, which may interfere with the use of the wearable device.

[0006] Therefore, when a wearable terminal is connected to a device via a wire and used by a user, it is necessary to provide a technology that enables good use of the wearable terminal.

[0007] According to a first aspect of the present invention, there is provided a wearable terminal as described below. The wearable terminal includes a display, a wired connection interface, and a processor. The display displays a virtual space that may include virtual objects, an image of real space and the virtual objects, or only an image of real space on a display screen. The wired connection interface is connected to a wired cable and is used for a wired connection with a device. The processor then notifies the user of the establishment of a wired connection with the device and, when it determines that the processor is located a predetermined distance or more away from the device, to warn the user about the use of the wearable terminal.

[0008] According to a second aspect of the present invention, there is provided an information processing method using a wearable device, the information processing method including: a display configured to display, on a display screen, a virtual space that may include virtual objects, an image of real space and virtual objects, or only an image of real space; a wired connection interface configured to be connected to a wired cable and used for a wired connection with a device; and a processor. The method includes the steps of: determining, by the processor, the establishment of a wired connection with the device; acquiring, by the processor, a distance from the device; and, when the processor identifies the establishment of a wired connection with the device and identifies that the wearable device is located at a predetermined distance or more from the device, issuing a notification to the user of the wearable device to warn them about use of the device.

[0009] According to the present invention, a technology is provided that enables a user to use a wearable device effectively when the wearable device is connected to a device via a wired connection. Note that other problems, configurations, and advantages will become clear from the description of the following embodiments of the invention.

[0010] FIG. 1 is a diagram for schematically illustrating an example of the appearance of a wearable terminal and an information processing system according to the present embodiment. FIG. 2 is a diagram for illustrating an example of a positional relationship between a wearable terminal, a wired connection destination device, and a wired connection medium in the wearable terminal and information processing system according to the present embodiment. FIG. 3 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 4 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 5 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 6 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 7 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 8 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 9 is a diagram for illustrating an example of a hardware configuration of a wearable terminal. FIG. 10 is a diagram for illustrating an example of information data and an example of a functional configuration of a wearable terminal. FIG. 11 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 12 is a diagram for illustrating an example of a display screen in use assistance control of a wired connection. FIG. 1 is a diagram for explaining an example of a display screen in use assistance control for a wired connection. FIG. 2 is a diagram for explaining an example of a display screen in use assistance control for a wired connection. FIG. 3 is a diagram for explaining an example of a display screen in use assistance control for a wired connection. FIG. 4 is a diagram for explaining an example of a display screen in use assistance control for a wired connection. FIG. 5 is a diagram for explaining an example of a display screen when a user is located inside a third distance. FIG. 6 is a flowchart for explaining an example of a processing flow of a wearable terminal or an information system. FIG. 7 is a diagram for explaining an example of a condition for starting use assistance control for a wired connection. FIG. 8 is a diagram for explaining an example of use assistance control for a wired connection. FIG. 9 is a diagram for explaining an example of a condition for ending use assistance control for a wired connection.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0013] Although various types of information may be described using expressions such as "table," "list," and "queue" as examples, the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0014] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0015] In some embodiments, processing performed by executing a program will be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0016] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] In the embodiment, a technology will be described that enables good use of a wearable terminal when a device is wired connected to the wearable terminal and the wearable terminal is used by a user by performing auxiliary control for wired connection use.

[0018] First, an example of a wearable terminal and an information system will be described with reference to Fig. 1. As shown in Fig. 1, in this embodiment, the wearable terminal 100 is an immersive HMD that is worn on the head of the user 10 so as to cover the eyes of the user 10. The wearable terminal 100 (in this example, the HMD) includes, for example, gaze sensors (101, 102), a camera 103, a distance measurement sensor 104, a positioning sensor 105, a microphone 106, a speaker 107, and a vibrator 108. Each component will be described in detail later.

[0019] The wearable terminal 100 can perform display operations based on, for example, a VR display mode in which a virtual space that may include virtual objects is displayed on the display screen, an AR display mode in which both an image of real space captured by the camera 103 and a virtual object are displayed on the display screen, or a captured image display mode in which only an image captured by the camera 103 is displayed on the display screen.

[0020] 1 , a character 151, which is an example of a virtual object, is displayed on the display screen 150a. The wearable terminal 100 can also display various types of information. In this example, information about the status of the wired connection of the wearable terminal 100 is displayed by the wired connection usage assistance control. For example, a message 152 indicating that the wearable terminal 100 is currently connected to the external device 120 via a wire, and a graphic 153 (an icon in this example) are displayed.

[0021] The type of information to be displayed is not limited to this example. Furthermore, the display format is not particularly limited. For example, the wearable terminal 100 may display information based on symbols and colors in addition to the message 152 and the graphic 153. Furthermore, the wearable terminal 100 may display information based on a combination of multiple types of the message 152, the graphic 153, the symbols, the colors, and the like.

[0022] Also provided is an information system including a wearable terminal 100 and a device 120 that is wired to the wearable terminal 100. In this example, the wearable terminal 100 includes a connection unit 109 to which a wired cable 130 is connected. As an example, the connection unit 109 can include a wired connection input / output interface, which will be described later. The device 120 includes a connection unit 121 to which the wired cable 130 is connected and that constitutes an input / output interface. The wearable terminal 100 and the device 120 are connected via the wired cable 130 (wired connection medium).

[0023] As an example, the device 120 is a monitor that outputs video processed by the wearable terminal 100. However, any device that is wired to the wearable terminal 100 may be used in the information system. The device 120 may be a charging device for the wearable terminal 100, an audio input device that is connected to the wearable terminal 100 and inputs external audio, an audio output device that outputs audio processed by the wearable terminal 100, a camera separate from the camera 103 of the wearable terminal 100, a personal computer, etc.

[0024] Also, as an example, the wired device 120 is placed on a table 122. However, the device 120 may be placed in a location in real space other than on the table 122. Note that the distance 140 shown in the figure is the distance between the device 120 and the user 10 in real space.

[0025] Next, an example of the usage status of a wearable device will be described with reference to Fig. 2A. Also, an example of a display will be described with reference to Figs. 2B to 2F. Fig. 2A shows an example of the positional relationship between a wearable device, a device connected to the wearable device via a wired connection, and a wired cable.

[0026] 2A , for example, a user 10 wearing an HMD 100 uses the HMD 100 at a predetermined distance (i.e., a distance corresponding to the distance 140) from a device 120 connected via a wired cable 130. Before performing use assistance control of the wired connection, for example, the user 10 inputs a first distance 201, a second distance 203, and a third distance 205 into the HMD 100, and the HMD 100 sets these distances (201, 203, 205). These distances (201, 203, 205) indicate the predetermined distance from the device 120 based on the position of the device 120.

[0027] For example, the first distance 201 is a distance set by the user 10 based on the judgment that a distance greater than this may cause inconvenience, such as being pulled by the connected wired cable 130. For example, the second distance 203 is a distance longer than the first distance 201 and shorter than the limit to which the wired cable 130 can be extended. For example, the third distance 205 is a distance shorter than the first distance 201 and closer to the device 120.

[0028] Therefore, the area 202 between the first distance 201 and the third distance 205 is an area where the user 10 is likely to feel no annoyance, such as being pulled by the wired cable 130, when moving. In contrast, the area 204 between the first distance 201 and the second distance 203 is an area where the user 10 is likely to feel annoyance, such as being pulled by the wired cable 130, when moving. The area inside the third distance 205 is an area near the device 120 (in this example, a monitor).

[0029] As an example, when the HMD 100 identifies that the distance between the HMD 100 and the device 120 is equal to or greater than the first distance 201 (i.e., when the HMD 100 identifies that the distance 140 is equal to or greater than the first distance 201), the HMD 100 may perform use assistance control for a wired connection. Here, for example, when the user 10 faces the device 120, the HMD 100 can display a wired connection area 211 in the field of view 206 of the camera of the HMD 100 when the user faces the device 120 in the use assistance control for a wired connection. Note that the wired connection area 211 relates to an area 204 between the first distance 201 and the second distance 203, as an example. Also, as an example, the wired connection area 211 relates to an area within the first distance 201, as an example. Also, as an example, the wired connection area 211 relates to an area within the second distance 203.

[0030] 2B shows an example of a display screen when a wired connection area is displayed. In the example of FIG. 2B, a character 151 is displayed on the display screen 210b. Then, the HMD 100 executes the use assistance control for the wired connection to display, for example, a wired connection area 211 within the field of view 206 on the display screen 210b. Note that the HMD 100 may display an image of real space corresponding to the wired connection area 211 acquired using the camera 103. The HMD 100 may also generate and display a virtual object representing the wired connection area 211.

[0031] On the other hand, for example, when the user 10 faces away from the device 120, the HMD 100 can display, in the use assistance control of the wired connection, a wired connection area 221 in the field of view 207 of the camera of the HMD 100 when the user 10 faces away from the device 120. Note that, similar to the wired connection area 211, the wired connection area 221 relates to the area 204 between the first distance 201 and the second distance 203, for example. Also, as an example, it relates to the area within the second distance 203.

[0032] 2C shows an example of a display screen when a wired connection area is displayed. In the example of FIG. 2C, a character 151 is displayed on the display screen 210c. Then, the HMD 100 executes the use assistance control for the wired connection to display, for example, a wired connection area 221 within the field of view 207 on the display screen 210c. Note that the HMD 100 may display an image of real space corresponding to the wired connection area 221 acquired using the camera 103. The HMD 100 may also generate and display a virtual object representing the wired connection area 221.

[0033] Therefore, by making the user 10 aware of the wired connection area (211, 221) that takes into account the length of the connected wired cable 130, it is possible to help prevent events that the user 10 finds annoying, such as the user 10 being pulled by the wired cable 130 (wired connection medium).

[0034] As an example, the HMD 100 may provide assistance by notification as a control for assisting use of the wired connection when it identifies that the HMD 100 is at a predetermined distance or more from the device 120. The HMD 100 can display, on the display screen, information regarding, for example, the direction toward the connection unit 121 of the device 120, the position of the connection unit 121 of the device 120, the distance between the HMD 100 or the user 10 and the device 120, and the length of the wired cable 130.

[0035] FIG. 2D shows an example of a display screen including notification information. In this example, a character 151 is displayed on the display screen 210d. The HMD 100 displays, as auxiliary controls for use of a wired connection, a direction mark 231 indicating the direction toward the monitor's connection port, a position mark 232 indicating the position of the monitor's connection port, a message 233 indicating the distance between the HMD 100 or the user 10 and the monitor, such as "192 cm," and a message 234 indicating the length of the wired cable 130, such as "wired cable length 200 cm." Note that real-space objects, such as the monitor, table, and wired cable shown with thin dashed lines, do not necessarily need to be actually displayed on the display screen 210d. In FIG. 2D, the real-space objects are illustrated to clarify their positions on the display screen.

[0036] By allowing the user 10 to recognize notification information such as the direction of the wired connected device 120, the position of the device 120, the distance between the device 120 and the HMD 100 or the user 10, and the length of the wired cable 130, it is possible to help prevent events that the user 10 finds annoying, such as the user 10 being pulled by the wired cable 130.

[0037] For example, the HMD 100 may provide assistance in notifying the user of the situation in real space as a use assistance control for a wired connection when it determines that the HMD 100 is at a certain distance or more from the device 120. The HMD 100 can display, for example, an object in real space captured by the camera 103 in the real space around the HMD 100, or a virtual object of this object, on the display screen.

[0038] 2E shows an example of a display screen. In this example, a character 151 is displayed on a display screen 210e. In use assistance control for a wired connection, the HMD 100 may display an object in real space captured by the camera 103, or a virtual object of this object. In this example, in use assistance control for a wired connection, the HMD 100 displays a table 122, a monitor which is an example of a device 120, and a wired cable 130.

[0039] By allowing the user 10 to recognize the situation in the real space, it is possible to help prevent events that the user 10 finds annoying, such as the user 10 being pulled by the wired cable 130.

[0040] As another example, the HMD 100 may provide assistance in notifying the user of the state of the wired cable 130 as a use assistance control for the wired connection. The HMD 100 may display the wired cable 130 on a display screen, for example.

[0041] 2F shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210f. In use assistance control for a wired connection, the HMD 100 may display an image of the wired cable 130 captured by the camera 103, or a virtual object 241 of the wired cable 130. Note that real-space objects such as a monitor and a table, shown by thin dashed lines, do not need to be actually displayed on the display screen 210f. In FIG. 2F, the monitor and the table are illustrated to clarify their positions on the display screen.

[0042] As shown in the figure, the HMD 100 may display the wired cable 130 or the virtual object 241 of the wired cable 130 on the display screen 210f so as to correspond to its position in real space. However, the HMD 100 may display the wired cable 130 or the virtual object 241 of the wired cable 130 on the display screen 210f so as not to correspond to its position in real space. For example, the HMD 100 may display the wired cable 130 or the virtual object 241 of the wired cable 130 on the edge of the display screen 210f so as not to interfere with the viewing of the user 10. Furthermore, for example, an area on the display screen 210f for displaying the wired cable 130 or the virtual object 241 of the wired cable 130 may be set in advance, and the HMD 100 may adjust the size of the displayed wired cable 130 or the virtual object 241 of the wired cable 130 to fit within this area.

[0043] Therefore, by allowing the user 10 to recognize the state of the wired cable 130 (for example, the shape of the wired cable, whether the wired cable is bent, whether the wired cable is tangled, etc.), it is possible to help prevent an event that the user 10 finds annoying, such as being pulled by the wired cable 130. Alternatively, it is possible to help the user 10 immediately notice an annoying event, such as being pulled by the wired cable 130, and take action.

[0044] Furthermore, with regard to notification of the state of the wired cable 130, the HMD 100 may display, for example, the images shown in FIGS. 2G-2H. FIG. 2G shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210g. The HMD 100 may also display information indicating that the wired cable 130 is taut and information indicating that pressure is being applied to the connection terminal during use assistance control for a wired connection. In this example, the HMD 100 displays, on the display screen 210g, a graphic 261 (icon) and a message 262 indicating that the wired cable 130 is taut, and a graphic 263 (icon) and a message 264 indicating that pressure is being applied to the connection terminal.

[0045] Therefore, by allowing the user 10 to recognize that the wired cable 130 is taut and that pressure is acting on the connection terminal, it is possible to help prevent an event that the user 10 finds annoying, such as being pulled by the wired cable 130. Alternatively, it is possible to help the user 10 immediately notice an annoying event, such as being pulled by the wired cable 130, and take action.

[0046] 2H shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210h. The HMD 100 may display information indicating that the wired cable 130 has been detached during use assistance control for a wired connection. In this example, a graphic 271 (icon) and a message 272 indicating that the wired cable 130 has been detached are displayed on the display screen 210h.

[0047] Therefore, by making the user 10 aware that the wired cable 130 has been disconnected, it is possible to assist the user 10 in using the HMD 100 .

[0048] Next, an example of the configuration of a wearable device will be described with reference to Fig. 3A. As described above, the wearable device 100 (e.g., HMD) includes gaze sensors (101, 102), a camera 103, a distance measurement sensor 104, a position measurement sensor 105, a microphone 106, a speaker 107, and a vibrator 108.

[0049] The gaze sensors (101, 102) include, for example, a left eye gaze sensor 101 and a right eye gaze sensor 102 for the right eye. The left eye gaze sensor 101 detects the left eye of the user 10 and can capture the viewpoint of the left eye at the gaze point by detecting the movement of the left eye, the movement of the gaze direction, etc. The right eye gaze sensor 102 detects the right eye of the user 10 and can capture the viewpoint of the right eye at the gaze point by detecting the movement of the right eye, the movement of the gaze direction, etc.

[0050] The wearable terminal 100 may perform predetermined processing using gaze information acquired from the gaze sensors (101, 102). The wearable terminal 100 may display an input operation screen such as an operation menu on the display 308. The wearable terminal 100 may acquire, as input operation information, the position of an icon, button, etc. on the input operation screen at which the user 10 is looking, and perform predetermined processing.

[0051] The process of detecting eye movement may utilize well-known techniques commonly used for eye tracking. For example, in a method using corneal reflex, infrared light is emitted from an infrared light-emitting diode (LED) toward the face, and an image is captured by an infrared camera, which is an example of an eye-gaze sensor. Then, using the light reflection point on the cornea as a reference point, eye movement (eye direction) can be estimated from changes in eye position and other geometric features.

[0052] The camera 103 captures a captured image by converting light incident through a lens into an electrical signal using an imaging element such as a CCD image sensor or a CMOS image sensor. The camera 103 can be positioned so as to capture an image of the surroundings of the wearable device 100. Based on the captured image captured by the camera 103, the wearable device 100 can acquire information such as the position of the wired connected device 120, the position of the connection portion 121 of the wired cable 130 in the wired connected device 120, and the position and state of the wired cable 130. Note that the placement, orientation, number, viewing angle, etc. of the cameras 103 are not particularly limited as long as appropriate processing can be performed.

[0053] The ranging sensor 104 is a sensor that can measure the distance from the wearable device 100 to an object, the object's position, and capture the three-dimensional shape of the object. Examples of the ranging sensor 104 include a LiDAR (Light Detection and Ranging) sensor that irradiates an object with laser light such as infrared light and measures the scattered light that bounces back; a TOF (Time of Flight) sensor that measures distance by measuring the reflection time of pulsed light irradiated on the object for each pixel; and a millimeter-wave radar that emits millimeter-wave radio waves and captures the reflected waves. The ranging sensor 104 may also be a sensor that performs measurements based on the angle at which light reflected by the object is received. That is, the ranging sensor 104 may be a triangulation sensor. The ranging sensor 104 may also be a stereo camera that performs measurements based on parallax images. The wearable terminal 100 processes the measurement results from the distance sensor 104 and can obtain information such as the position of the device 120 to which it is connected via a wired connection, the position of the connection part 121 of the wired cable 130 in the device 120 to which it is connected via a wired connection, and the position and status of the wired cable 130.

[0054] The positioning sensor 105 is, for example, a GPS (Global Positioning System) receiver. In this case, by receiving satellite radio waves from GPS satellites and performing positioning calculations, it is possible to detect, for example, the coordinates on the Earth at which the wearable device 100 is located. The positioning calculations may be performed by the GPS receiver, or by other components (such as the processor 320 of the wearable device 100 or an external device). The wearable device 100 may acquire location information using other methods. For example, the wearable device may determine its current location by analyzing images captured by the camera 103 and / or three-dimensional point cloud data measured by the ranging sensor 104; techniques such as SLAM (Simultaneous Localization and Mapping) are known. When acquiring location information without using the positioning sensor 105, the positioning sensor 105 may be omitted.

[0055] Also, known technologies include a visual positioning system (VPS) that identifies a location by analyzing camera images. The wearable device 100 may acquire location information by the VPS using images captured by the camera 103. The wearable device 100 may also transmit the images captured by the camera 103 to an external device. The external device may then acquire location information based on the VPS using the captured images and distribute the acquired location information to the wearable device 100. Note that when acquiring location information without using the positioning sensor 105, the positioning sensor 105 may be omitted.

[0056] Furthermore, the wearable device 100 may acquire location information by Pedestrian Dead Reckoning (PDR) using, for example, a sensor described below. When acquiring location information without using the positioning sensor 105, the positioning sensor 105 may be omitted.

[0057] The microphone 106 collects the user's own speech, external sounds, and the like, and converts them into audio data. The user 10 may speak to give instructions such as input operations, and the wearable device 100 may acquire the audio data collected by the microphone 106 as instruction information for input operations and the like, and execute predetermined processing.

[0058] The speaker 107 outputs sound based on the audio data, and can, for example, notify the user 10 of notification information by sound. The wearable terminal 100 can, for example, issue a warning message such as "Connected via wire!" by sound from the speaker 107 to notify the user 10. By providing notifications by sound, usability can be improved.

[0059] The vibrator 108 is a device that generates vibrations under the control of the processor 320, and converts notification information sent by the wearable terminal 100 to the user 10 into vibrations. The vibrator 108 transmits vibrations to the user 10 wearing the wearable terminal 100, and can notify the user 10 of the sending of notification information such as a warning message such as "Connected via wired connection!" or the display operation of the wired connection area, thereby improving usability.

[0060] The wearable terminal 100 also includes an acceleration sensor 304, a gyro sensor 305, and a geomagnetic sensor 306. The acceleration sensor 304 can detect, for example, the tilt and direction of the wearable terminal 100. The gyro sensor 305 is a sensor that detects angular velocity or angular acceleration in the rotational direction and can capture the posture of an object. Therefore, the acceleration sensor 304 and the gyro sensor 305 can be used to detect the posture, such as the tilt and direction, of the wearable terminal 100. The geomagnetic sensor 306 is a sensor that detects the Earth's magnetic force and detects the direction in which the wearable terminal 100 is facing. It is also possible to detect the movement of the wearable terminal 100 by using a three-axis type sensor that detects geomagnetic fields in the up-down direction in addition to the front-back and left-right directions and captures changes in geomagnetic field in response to the movement of the wearable terminal 100.

[0061] Using these sensors, the wearable terminal 100 can acquire information about the posture and movement of the user 10. The wearable terminal 100 can then use the acquired information to grasp the state of the user 10, and as a result, can analyze with higher accuracy the position and state of the wired connected device 120, the position and state of the wired cable 130, and the like.

[0062] The wearable terminal 100 also includes an operation input interface 307 and a display 308. The operation input interface 307 is an interface that realizes input means such as gaze, a hand, a pointer, or even a keyboard, key buttons, touch keys, or the like, which are operation devices separate from the wearable terminal 100, and the user 10 inputs desired information via the operation input interface 307.

[0063] The wearable terminal 100 may display an input operation screen such as an operation menu on the display screen of the display 308, acquire the position on the input operation screen at which the user 10 is looking as input operation information, and perform an operation according to the input operation information. The wearable terminal 100 may also display a cursor such as a hand or a pointer on the input operation screen, acquire an operation by the user 10 using the cursor as input operation information, and perform an operation according to the input operation information. The wearable terminal 100 may also acquire a gesture by the user 10 as input operation information and perform an operation according to the input operation information. In addition, when input is performed using an input operation device such as a keyboard, key buttons, or touch keys, the input operation device may be connected to the wearable terminal 100 main body using a wired cable 130 or may be connected wirelessly.

[0064] Examples of operation inputs by the user 10 include setting the first, second, and third distances (201, 203, 205) related to the use assistance control of a wired connection, initial setting of the contents of warning messages (152, 234, 262, 264, 272), setting to cancel the control when a wired connection is used, etc. Further, examples of operation inputs by the user 10 include various setting operations such as application operations and volume adjustment.

[0065] The display 308 can be configured using a display such as a liquid crystal display, a touch panel display, or an organic EL display, and displays virtual objects, camera-captured images, etc. For example, if the wearable device 100 is an immersive HMD, the display 308 is configured using a liquid crystal panel, a display that displays holographic images, etc., and displays virtual objects, camera-captured images, etc.

[0066] The wearable terminal 100 also includes a processor 320, a memory 330, a wired connection input / output interface 345, a short-range wireless communication interface 346, and a network communication interface 347. The processor 320 is a main component that executes predetermined processes and is configured using a CPU (Central Processing Unit) and the like. As shown in the figure, each component of the wearable terminal 100 is connected to the processor 320 via a bus 350. The processor 320 executes an operating system (OS) 333 and application programs 334 for operation control and the like stored in the memory 330, thereby controlling each component of the wearable terminal 100 and realizing functions of the OS, middleware, applications, and the like.

[0067] The memory 330 is composed of a main storage device, a non-volatile storage device, etc., and stores various programs 331 and information data 332 handled by the processor 320, etc. An example of the information data 332 will be described later. The memory 330 stores, as the programs 331, for example, an operating system 333 and various application programs 334 for controlling operations used by the processor 320. The memory 330 may also store various setting contents, contents set by the user 10, etc.

[0068] The wired connection input / output interface 345 (wired connection interface) is a component used for a wired connection, and is connected to the wired connection destination device 120 via the wired cable 130. In the figure, the wired connection destination device 120 is, for example, an information processing device 360 ​​(wired connection destination device), and the wired connection input / output interface 345 is an input / output interface that performs wired communication of information with the wired connection destination information processing device 360 ​​via the wired cable 130. Note that a charging device or the like may be connected as the wired connection destination device 120, and the wired connection input / output interface 345 may be an input / output interface that inputs or outputs power via the wired cable 130. The wired connection input / output interface 345 may also be configured to be connected via Power of Ethernet (PoE; Ethernet is a registered trademark).

[0069] Examples of devices 120 that can be connected via a wired connection using the wired connection input / output interface 345 include display devices such as a monitor, information processing devices such as a personal computer, and charging devices as exemplified in FIG. 1, as well as audio devices such as earphones or loudspeakers, and input operation devices such as a keyboard, key buttons, and touch keys.

[0070] The wired cable 130 is a connection cable to the device 120 that is the wired connection destination. More specifically, the wired cable 130 is a connection cable such as a charging cable used to connect to a charging device, a twisted pair cable used for POE, a cable for wired earphones, or a cable for audio. If the information processing device 360 ​​is a personal computer, the wired cable 130 is, for example, a LAN cable.

[0071] The short-range wireless communication interface 346 is a communication interface that performs short-range wireless communication with a device (in this example, the information processing device 360) within a range where short-range wireless communication is possible. Short-range wireless communication is performed using, for example, an electronic tag, but is not limited to this. If the wearable terminal 100 is located near a device that is at least capable of wireless communication, short-range wireless communication may be performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or a wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g).

[0072] The network communication interface 347 is a communication interface that performs wireless communication with an external information server 380 or the like via an external network 381 using base station communication or the like. The network communication interface 347 can receive various information stored in the information server 380. Note that as base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access) or GSM (Registered Trademark) (Global System for Mobile communications) can be used.

[0073] Next, an example of information data and an example of the functional configuration of a wearable terminal will be described with reference to FIG. 3B.

[0074] As shown in FIG. 3B , the memory 330 can store, as information data 332, image information 335, ranging information 336, location information 337, wired connection area information 338, virtual space information 339, wired connection medium status information 340, wired connection status information 341, and video information 342, as examples.

[0075] The image information 335 is information about an image acquired by the wearable terminal 100, and includes, for example, information about an image of real space acquired by the camera 103 and information about an image acquired by the distance measurement sensor 104. Note that the image information 335 may also include information about an image used to acquire position information of the wearable terminal 100.

[0076] The ranging information 336 is information relating to the distance to an object around the wearable device 100, and includes, for example, information relating to the distance to the object measured by the ranging sensor 104. The ranging information 336 may also include information relating to the position of the object acquired using the ranging sensor 104. The ranging information 336 may also include information relating to the shape of the object acquired using the ranging sensor 104.

[0077] The location information 337 is information about the location of the wired connected device 120 and information about the location of the wearable terminal 100 (i.e., the location of the user 10). The location information 337 is acquired by a wired connection destination device / wearable terminal location information processing unit 321, which will be described later.

[0078] The wired connection area information 338 is information about a wired connection area, which is an area where a connected wired cable 130 may be present. As an example, as described in the above example display screen, the wired connection area information 338 may include information about an area between the first distance 201 and the second distance 203, which is an area where the wired cable 130 is likely to be an annoyance. Furthermore, the wired connection area information may include information about an area within the first distance 201. Furthermore, as an example, the wired connection area information may include information about an area within the second distance 203. Furthermore, as an example, the wired connection area information 338 may include information about an area between the first distance 201 and the third distance 205. The wired connection area information 338 is acquired by the wired connection area setting processing unit 323, which will be described later.

[0079] The virtual space information 339 is information about the virtual space and virtual objects displayed by the wearable terminal 100. The virtual space information 339 may be generated by the processor 320 of the wearable terminal 100 executing a program, application software, or the like. Alternatively, the virtual space information 339 may be information generated by an external device (for example, an information server 380) and acquired by the wearable terminal 100 through communication.

[0080] The wired medium status information 340 is information relating to the status of the wired cable 130. The wired medium status information 340 is acquired by the wired medium status identification processing unit 326, which will be described later.

[0081] The wired connection status information 341 is information indicating whether the wired cable 130 is connected to the device 120. In other words, the wired connection status information 341 indicates whether the wired cable 130 is disconnected. The wired connection status information 341 is acquired by the wired connection status identification processing unit 327, which will be described later.

[0082] Video information 342 is information relating to video acquired using camera 103. Video information 342 includes video captured by camera 103 of connection unit 121 of device 120 that is the wired connection destination. This video is a video showing the disconnection of wired cable 130, and includes, for example, a scene in which wired cable 130 is disconnected, a scene when connected wired cable 130 is disconnected, and the like.

[0083] The timing for starting acquisition of video information may be, for example, the timing when the wired cable 130 is detached. That is, the wearable terminal 100 may start capturing this video (playback video) when the wired cable 130 is detached. Furthermore, the timing for acquiring video information may be, for example, the timing when the wired cable 130 reaches a predetermined state. That is, the wearable terminal 100 may start capturing this video (playback video) when, for example, information is acquired that a predetermined amount of pressure or more has been applied to the wired cable 130, the connection terminal of the wiredly connected device 120, and / or the connection terminal of the wearable terminal 100. Note that the wearable terminal 100 may stop recording a predetermined time after the wired cable 130 is detached from the connection terminal.

[0084] 3B , the wearable terminal 100 includes, as an example, a wired connection destination device / wearable terminal position information processing unit 321, an icon / message notification processing unit 322, a wired connection area setting processing unit 323, a captured image display processing unit 324, a virtual space generation and display processing unit 325, a wired connection medium status identification processing unit 326, a wired connection status identification processing unit 327, and a video generation and display processing unit 328. These components are realized by the processor 320 storing data such as programs in the main storage device of the memory 330 and processing the data. Note that other components (sensors, interfaces, etc.) may also be used to realize these components.

[0085] The wired connection destination device / wearable terminal position information processing unit 321 acquires the position of the wired connection destination device 120 and the position of the wearable terminal 100. The wired connection destination device / wearable terminal position information processing unit 321 acquires the position of the wired connection destination device 120, for example, by using image information 335 and / or distance measurement information 336. Furthermore, the wired connection destination device / wearable terminal position information processing unit 321 may acquire the position of the wired connection destination device 120 by, for example, communicating with the wired connection destination device 120.

[0086] The wired connection destination device / wearable terminal position information processing unit 321 may, for example, acquire the position of the wearable terminal 100 using the positioning sensor 105. The wired connection destination device / wearable terminal position information processing unit 321 may also acquire the position of the wearable terminal 100 using, for example, a technology related to self-position estimation, a VPS technology, or the like.

[0087] The wired connection destination device / wearable terminal location information processing unit 321 can store information about the location of the wired connection destination device 120 and the location of the wearable terminal 100 in the memory 330 as location information 337 .

[0088] For example, in the case of wired connection assistance control, the icon / message notification processor 322 generates an icon and / or a message of notification information and outputs it to the display 308 to notify the user 10. The icon / message notification processor 322 may generate audio data related to the message and output the audio data to notify the user 10 of the content based on the notification information. The icon / message notification processor 322 may also generate vibration data corresponding to the content of the message and vibrate the vibrator 108 based on the generated vibration data to notify the user 10 of the content based on the notification information.

[0089] The wired connection area setting processing unit 323 sets a wired connection area based on information about the set distances (201, 203, 205), and stores the set content in the memory 330 as wired connection area information 338. The wired connection area setting processing unit 323 sets an area within the second distance 203 as the wired connection area information 338, and stores the wired connection area information 338 in the memory 330.

[0090] When displaying a virtual object or the like in an AR display mode, the captured image display processing unit 324 uses the image information 335 to generate an image captured by the camera 103 as a background and displays it on the display 308. Furthermore, when performing a captured image display mode in which only images captured by the camera 103 are displayed on the display screen, the captured image display processing unit 324 uses the image information 335 to display the image captured by the camera 103 on the display 308.

[0091] Furthermore, in the use assistance control of the wired connection, the captured image display processing unit 324 uses the image information 335 to display images of surrounding objects (such as the wired cable 130) on the display 308.

[0092] When displaying a virtual object or the like in an AR display format, the virtual space generation and display processing unit 325 uses the virtual space information 339 to generate the virtual object to be displayed and displays it on the display 308. When displaying a virtual space, a virtual object, or the like in a VR display format, the virtual space generation and display processing unit 325 uses the virtual space information 339 to generate the virtual space, the virtual object, or the like to be displayed and displays it on the display 308.

[0093] Furthermore, in the use assistance control for a wired connection, the virtual space generation and display processing unit 325 generates virtual objects of surrounding objects (such as the wired cable 130) using the image information 335 or the like, and displays the virtual objects on the display 308. In the use assistance control for a wired connection, the virtual space generation and display processing unit 325 generates virtual objects such as direction marks and position marks, and displays the virtual objects on the display 308.

[0094] The wired connection medium state identification processing unit 326 acquires wired connection medium state information 340 relating to the state of the wired cable 130 and stores it in the memory 330. The wired connection medium state identification processing unit 326 uses, for example, image information 335, distance measurement information 336, etc. to identify the shape of the wired cable 130, detect whether the wired cable 130 is taut, whether the wired cable 130 is slack, whether the wired cable 130 is tangled, etc., and acquires the wired connection medium state information 340. Furthermore, the wired connection medium state identification processing unit 326 may compare the acquired information about the length of the wired cable 130 with information about the distance between the wired connection destination device 120 and the wearable terminal 100 or the user 10 to detect whether the wired cable 130 is taut, whether the wired cable 130 is slack, etc., and acquire the wired connection medium state information 340.

[0095] In addition, the wired connection medium status identification processing unit 326 may acquire information regarding whether pressure is being applied to the wired cable 130, the connection terminal of the wired connected device 120, and / or the connection terminal of the wearable terminal 100, and store the information in memory 330.

[0096] As an example, the pressure at each connection terminal is detected by a piezoelectric element incorporated in the connection terminal. The wearable terminal 100 acquires the pressure at the connection terminal on the device 120 side, for example, via the wired cable 130. The wearable terminal 100 also acquires the pressure at the connection terminal on the device 120 side, for example, by communication with the device 120 (wired communication or wireless communication). The wearable terminal 100 acquires the pressure on the wearable terminal 100 side from the connection terminal of the wearable terminal 100. The pressure on the wired cable 130 can be, for example, the same as the pressure applied to the connection terminal. Alternatively, a piezoelectric element may be provided in the connection terminal of the wired cable 130, and the wearable terminal 100 may acquire the pressure applied to the connection terminal of the wired cable 130 from the piezoelectric element.

[0097] The wired connection status identification processing unit 327 acquires wired connection status information 341 related to the attachment and detachment of the wired cable 130 and stores the information in the memory 330. For example, the wired connection status identification processing unit 327 may identify the presence or absence of input / output of data, etc., via the wired connection input / output interface 345, and acquire the wired connection status information 341. That is, for example, when input / output of data, signals, power, etc., is not possible between the wearable terminal 100 and the device 120 via the wired connection input / output interface 345, the wired connection status identification processing unit 327 may acquire wired connection status information 341 indicating that the wired connection is disconnected.

[0098] Furthermore, the wired connection status identification processing unit 327 may use image information 335, distance measurement information 336 indicating the shape of an object, and the like, around the connection unit 121 to identify whether the wired cable 130 is connected, and acquire wired connection status information 341. Furthermore, the wired connection status identification processing unit 327 may acquire information regarding whether a wired connection has been established from the device 120 via communication.

[0099] The moving image generation and display processing unit 328 performs processing to output a moving image based on the moving image information 342. For example, in the use assistance control of a wired connection, the moving image generation and display processing unit 328 uses the moving image information 342 to generate a moving image including, for example, a scene in which the wired cable 130 is disconnected and / or a scene when the connected wired cable 130 is disconnected, and outputs the moving image to the display 308.

[0100] Next, examples of displays in various usage situations will be described with reference to Figures 4 to 7A. Figures 4 to 7A are diagrams showing examples of display screens in use assistance control for wired connections.

[0101] In the example of FIG. 4 , the portion of the wired cable 130 surrounded by the encircling line 401 is in a state where it is outside the first distance 201, a virtual object of a character 151 is displayed on the display screen 150b, and the user 10 is using the HMD 100 in the area 202 between the first distance 201 and the third distance 205. Even when the user 10 is located in the area 202 between the first distance 201 and the third distance 205, a portion of the wired cable 130 is located outside the first distance 201. Therefore, in the use assistance control for the wired connection, the HMD 100 displays a message and / or an icon indicating that a wired connection is in progress on the display screen 150b, for example, by executing the icon / message notification processing unit 322. For example, the display screen 150b displays or re-displays an icon 153 indicating that a wired connection is in progress. Note that the icon 153 may be highlighted based on factors such as color or size. Also, for example, a warning message 152 saying "Wired connection in progress!" is displayed on the display screen 150b.

[0102] This can prevent the user 10 from feeling annoyed by being pulled by the wired cable 130 that protrudes into an area outside the first distance 201.

[0103] 5 , the user 10 is facing away from the wired device 120 (in this example, a monitor), and a character virtual object 151 is displayed on the display screen 210i. In use assistance control for a wired connection, the HMD 100 may display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. However, as shown in FIG. 5 , when the user 10 faces away from the device 120, the wired cable 130 may not be within the field of view 501 of the user 10, and the wired cable 130 or a virtual object of the wired cable 130 may not be displayed on the display screen so as to correspond to a position in real space.

[0104] 5 , the HMD 100 may not display the wired cable 130 or a virtual object of the wired cable 130 to correspond to a position in real space during use assistance control for a wired connection. In such a situation, the HMD 100 may execute the virtual space generation and display processing unit 325 to generate a virtual object 502 of the wired cable 130 and display the virtual object 502 of the wired cable 130 at an appropriate position on the display screen 210i. The display position may be, for example, on the edge of the display screen 210i so as not to interfere with the user 10's viewing of content, etc. Furthermore, the HMD 100 may generate and display a virtual object 502 of the entire wired cable 130, or may generate a virtual object 502 of a portion of the wired cable 130 (e.g., a portion nearby and outside the field of view) and display it on the display screen 210i.

[0105] As a result, even if the actual wired cable 130 does not appear on the display screen within the user's field of view, the user 10 can recognize the presence of the wired cable 130 as a virtual object. This allows the user 10 to sense an annoyance caused by, for example, being pulled by the wired cable 130, and can prevent this from occurring.

[0106] 6A , the user 10 is using the HMD 100 with the distance between the device 120 (in this example, a monitor) and the HMD 100 increased and the wired cable 130 taut. In the use assistance control of the wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 603 of the wired cable 130 by executing the virtual space generation and display processing unit 325 and displays it on the display screen 210j.

[0107] Furthermore, the HMD 100 executes the virtual space generation and display processing unit 325 to generate and display virtual objects, such as a direction mark 601 indicating the direction toward the connection unit 121 of the device 120 and a position mark 602 indicating the position of the connection unit 121 of the device 120. Furthermore, the HMD 100 executes the icon / message notification processing unit 322 to display a message 262 and / or an icon 261 indicating that the wired cable 130 is taut on the display screen 210j. For example, the display screen 210j displays or re-displays the icon 261 indicating that the wired cable 130 is taut. Note that the icon 261 may be highlighted based on color, size, or the like. For example, the display screen 210j displays a warning message 262 stating, "The wired cable is taut!"

[0108] This allows the user 10 to recognize that the wired cable 130 is taut. This helps the user 10 to immediately notice and deal with any annoying events, such as being pulled by the wired cable 130. Note that the table and equipment (monitor in this example) in the display screen 210j shown in the figure are shown to clarify their positions on the display screen 210j, and may not actually be displayed.

[0109] 6B , the user 10 is using the HMD 100 with the wired cable 130 caught on the object 610 and pressure acting on the connection terminal. In the use assistance control for the wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 executes the virtual space generation and display processing unit 325 to generate a virtual object 604 of the wired cable 130 caught on the object 610 and displays it on the display screen 210k.

[0110] 6A , the HMD 100 executes the virtual space generation and display processing unit 325 to display a direction mark 601 pointing toward the device 120 and a position mark 602 indicating the connection unit 121 of the device 120, and executes the icon / message notification processing unit 322 to display an icon and / or a message. For example, the display screen 210k displays or re-displays an icon 263 indicating that pressure is being applied to the connection terminal. Note that the icon 263 may be highlighted based on color, size, or the like. Also, for example, the display screen 210k displays a warning message 264 stating, "Pressure is being applied to the connection terminal!"

[0111] This allows the user 10 to recognize that pressure is being applied to the connection terminal. This allows the user 10 to sense an annoying event, such as being pulled by the wired cable 130, and prevent the event from occurring. Alternatively, this can help the user 10 to immediately sense an annoying event, such as being pulled by the wired cable 130, and take appropriate action. Note that the table, equipment (in this example, a monitor), and objects on the display screen shown in the figure are illustrated to clarify their positions on the display screen, and may not actually be displayed.

[0112] In the example of FIG. 6C , the user 10 is using the HMD 100 with a tangled portion 605 in the wired cable 130. In the use assistance control of a wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 607 of the wired cable 130 by executing the virtual space generation and display processing unit 325 and displays it on the display screen 2101. Note that this virtual object 607 includes a tangled portion 606 that corresponds to the tangled portion 605 in real space. Furthermore, the HMD 100 may display an icon and / or a message indicating that the wired cable 130 is tangled by executing, for example, the icon / message notification processing unit 322.

[0113] This allows the user 10 to recognize that the wired cable 130 has become tangled. The user 10 can then sense an annoyance caused by, for example, being pulled by the wired cable 130, and can prevent such an incident from occurring. Note that the table and devices (monitors in this example) on the display screen shown in the figure are shown to clarify their positions on the display screen, and may not actually be displayed.

[0114] 6C , the wearable terminal 100 (e.g., HMD) may identify the tangled portion 605 of the wired cable 130, and identify the state of the wired cable 130 excluding that portion in the wired connection medium state identification processing unit 327. For example, if the wired cable 130 is tangled and is taut, the HMD 100 may identify the tangled portion 605 of the wired cable 130, and identify the state of the wired cable 130 as being straight in the wired connection medium state identification processing unit 327.

[0115] 7A shows an example of a situation in which the wired cable 130 connected to the device 120 becomes disconnected while the user 10 is using the HMD 100 at a position greater than the third distance 205. In the use assistance control for the wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 703 of the wired cable 130 by executing the virtual space generation and display processing unit 325 and displays it on the display screen 210m.

[0116] Furthermore, the HMD 100 executes the virtual space generation and display processing unit 325 to display a direction mark 701 pointing toward the device 120 and a position mark 702 indicating the connection portion 121 of the device 120, and executes the icon / message notification processing unit 322 to display an icon and / or a message. For example, an icon 271 indicating that the wired cable 130 has been disconnected is displayed or re-displayed on the display screen 210m. Note that the icon 271 may be highlighted based on color, size, or the like. Also, for example, a warning message 272 stating "The wired cable has been disconnected!" is displayed on the display screen 210m.

[0117] This allows the user 10 to recognize that the wired cable 130 has been disconnected. Note that the table and equipment (monitor in this example) on the display screen 210m shown in the figure are shown to clarify their positions on the display screen 210m, and may not actually be displayed.

[0118] Furthermore, the HMD 100 may display on the display screen a video (playback video) showing the detachment of the wired cable 130. In the use assistance control of the wired connection, the HMD 100 can display on the display screen 210n a video 704 including the situation in which the wired cable 130 has been detached by executing the video generation and display processing unit 328. By viewing the video 704, the user 10 can recognize the situation in which the wired cable 130 has been detached.

[0119] The user 10 may set a video playback time, which is the time from the start to the end of the video, and the video playback display processor 328 may display the video 704 showing the situation in which the wired cable 130 has been disconnected, for the video playback time set by the user 10. Alternatively, the video playback time may be determined automatically, and the video playback display processor 328 may display the video 704 showing this situation for the automatically determined video playback time. In the example of FIG. 7A , the video generation and display processor 328 generates a 15-second video 704 including the situation in which the wired cable 130 has been disconnected, based on the video playback time. The video 704 displays objects around the connection portion 121 of the device 120, and in the example of the display screen 210 n in FIG. 7A , a table, the device 120 (in this example, a monitor), and the wired cable 130 are displayed.

[0120] The display method is not particularly limited. For example, the user 10 may select a display related to the display screen 210m or a display related to the display screen 210n, input the selection, and the wearable terminal 100 (in this example, the HMD) may display the display screen 210m or the display screen 210n based on the user's selection. Alternatively, the wearable terminal may automatically transition between the display screen 210m and the display screen 210n. The wearable terminal 100 may simultaneously display the display screen 210m and the display screen 210n on the display 308.

[0121] These display screens (210m, 210n) allow the user 10 to accurately recognize the problem when a problem occurs with a wired connection, for example, and receive assistance in easily performing work to correct the problem.

[0122] Next, with reference to FIG. 7B , an example of processing when the user 10 is located inside the third distance 205 will be described. FIG. 7B shows an example of a display screen when the user 10 is located inside the third distance 205. If the wired cable 130 becomes unplugged while the user 10 is located within the third distance 205, it is considered that the user 10 intentionally unplugged the wired cable 130 from the device 120. Therefore, the wearable terminal 100 (in this example, the HMD) does not display information indicating that the wired cable 130 has been unplugged on the display screen 210o. The third distance 205 is a criterion for determining that the user 10 intentionally unplugged the wired cable 130.

[0123] Next, an example of a processing flow of the wearable terminal 100 or the information system will be described with reference to Figure 8. The processor 320 of the wearable terminal 100 confirms the establishment of a wired connection between the wearable terminal 100 and the device 120 (S801). Then, while the user 10 is using the wearable terminal 100, the processor 320 determines whether the execution conditions for wired connection usage assistance control (wired connection usage assistance control) are met (S802). Then, if the processor 320 determines that the execution conditions for wired connection usage assistance control are met, it executes wired connection usage assistance control (S803).

[0124] 9A , an example of a determination item related to S802 will be described. The processor 320 performs wired connection usage assistance control on the condition that, for example, the distance between the wearable terminal 100 and the device 120 (apparatus) connected via wired connection or the connection unit 121 (wired connection point) of the device 120 (i.e., the distance 140 between the device 120 and the user 10) is equal to or greater than a first distance 201 (condition 1).

[0125] The processor 320 performs the assistive control of the wired connection, for example, on the condition that at least a portion of the wired cable 130 (wired connection medium) is located at a position equal to or greater than the first distance 201 from the device 120 to which the wired connection is connected or the connection unit 121 of the device 120 (Condition 2). For example, when a portion of the wired cable 130 is located outside the first distance 201, the processor 320 performs the assistive control of the wired connection.

[0126] The processor 320 performs auxiliary control of the use of the wired connection, for example, on the condition that the second distance 205 is greater than the length of the wired cable 130 (condition 3).

[0127] The processor 320 performs the wired connection usage assistance control, for example, on the condition that the pressure applied to the connection terminal of the wired cable 130 is equal to or greater than a first pressure amount (Condition 4). Here, the first pressure amount is a pressure amount that is set appropriately.

[0128] The processor 320 performs the assistive control of the wired connection, for example, on the condition that the pressure applied to the connection terminal of the wired device 120 is equal to or greater than a second pressure amount (Condition 5). Here, the second pressure amount is an amount of pressure that is set appropriately.

[0129] The processor 320 performs the wired connection usage assistance control, for example, on the condition that the pressure applied to the connection terminal of the wearable terminal 100 is equal to or greater than a third pressure amount (Condition 6). Here, the third pressure amount is a pressure amount that is set appropriately.

[0130] The processor 320 performs auxiliary control of the wired connection, for example, on the condition that the wired cable 130 is in a straight line (Condition 7). Here, the processor 320 can determine the state of the wired cable 130 using, for example, the wired connection medium information 340.

[0131] The processor 320 performs usage assistance control for the wired connection, for example, when the distance between the wearable terminal 100 and the device 120 to which the wearable terminal 100 is wiredly connected, or the connection unit 121 of the device 120, is equal to or greater than the third distance 205, and the wired connection has been disconnected (condition 8). That is, the processor 320 performs usage assistance control for the wired connection, when the distance 140 between the device 120 and the user 10 is equal to or greater than the third distance 205, and the wired connection has been disconnected. Here, the processor 320 can determine the disconnection of the wired connection, for example, by using the wired connection status information 341.

[0132] 9B, an example of the control content related to step S803 will be described. That is, an example of the control content of the usage assistance control for the wired connection will be described. Note that the processor 320 may execute a combination of multiple control contents.

[0133] The processor 320 issues a notification indicating that a wired connection is in progress (control 1), for example. Here, the processor 320 confirms that a wired connection is in progress, for example, based on the wired connection status information 341. The manner of notification is not particularly limited, and the processor 320 issues a notification to that effect by displaying an icon, a message, a virtual object, outputting a sound, outputting a vibration, or the like.

[0134] The processor 320 displays, for example, the wired connection destination device 120 and / or an area within the second distance 203 from the device 120 (control 2). Here, the processor 320 generates and displays a virtual object indicating the wired connection area, for example, based on the wired connection area information 338. Note that the processor 320 may also display, for example, an image of real space corresponding to the wired connection area acquired using the camera 103.

[0135] The processor 320 issues a warning that the set second distance 203 is longer than the length of the wired cable 130 (control 3), for example. Note that the length of the wired cable 130 used to connect to the device 120 may be input in advance by the user 10, for example. The processor 320 may then store the length of the wired cable 130 set by the user 10 in the memory 330 as information data 332 and use it in processing.

[0136] The processor 320 displays, for example, a direction mark indicating a direction from the wearable device 100 to the device 120 connected by wire or the connection unit 121 of the device 120 (control 4). Here, the processor causes the virtual space generation and display processing unit 325 to generate and display a virtual object of the direction mark.

[0137] The processor 320 displays, for example, the wired device 120 or the connection part 121 of the device 120 (control 5). The processor 320 displays, for example, the wired device 120 or the connection part 121 of the device 120 acquired by the camera 103. The processor 320 generates and displays, for example, a virtual object of a mark indicating the wired device 120 or the connection part 121 of the device 120 by the virtual space generation and display processing unit 325.

[0138] The processor 320 displays, for example, the distance from the wearable terminal 100 to the device 120 connected via a wired connection or to the connection unit 121 of the device 120 (control 6). Here, the processor 320 displays the distance based on, for example, the display of an icon, a message, or a virtual object. The processor 320 may also provide, for example, a voice notification regarding the distance.

[0139] The processor 320 displays, for example, the length of the currently connected wired cable 130 (control 7). Here, similar to control 3 above, the processor 320 may store the length of the wired cable 130 set by the user 10 in the memory 330 as information data 332 and use it for processing.

[0140] The processor 320 issues a notification indicating that the wired cable 130 is straight (control 8), for example. That is, the processor 320 issues a notification indicating that the wired cable 130 is taut, for example. Here, the processor 320 checks the shape of the wired cable 130 based on the wired connection medium status information 340, for example. The manner of notification is not particularly limited, and the processor 320 issues the notification by displaying an icon, a message, a virtual object, outputting sound, outputting vibration, or the like.

[0141] The processor 320 issues a notification indicating that pressure is being applied to, for example, at least one connection terminal of the wired cable, the wearable terminal, or the wired connected device (control 9). Here, the processor 320 acquires information indicating that pressure is being applied, for example, based on the wired connection medium status information 340. The manner of notification is not particularly limited, and the processor 320 issues the notification by displaying an icon, a message, a virtual object, outputting audio, outputting vibration, or the like.

[0142] The processor 320 displays, for example, an image of the real space (control 10). Here, the processor 320 may, for example, confirm that the wired cable 130 is within the field of view of the camera 103 and display an image of the real space including the wired cable 130. The processor 320 may also, for example, confirm that the wired connection destination device 120 is within the field of view of the camera 103 and display an image of the real space including the wired connection destination device 120.

[0143] The processor 320 generates and displays, for example, a virtual object corresponding to the connected wired cable 130 (control 11). Note that, for example, if the wired cable 130 is tangled, a virtual object of the wired cable 130 including the tangle may be displayed. The processor 320 may also display an image of the wired cable 130 acquired by the camera 103.

[0144] The processor 320 displays, for example, the position of the disconnected wired cable 130 (control 12). The processor 320 generates and displays, for example, a virtual object indicating the position of the connection portion 121 of the device 120 to which the wired cable 130 was connected. The processor 320 may also generate and display a virtual object indicating the position of the end of the disconnected wired cable 130. The processor 320 may also provide a notification indicating that the cable has been disconnected. The manner of the notification is not particularly limited, and the processor 320 may provide the notification indicating that the cable has been disconnected by displaying an icon, a message, a virtual object, outputting a sound, outputting a vibration, or the like.

[0145] The processor 320 displays, for example, a playback video (control 13). That is, the processor 320 displays, for example, a video relating to a situation in which the wired cable 130 is disconnected.

[0146] 8 , the description of the processing flow will be continued. The processor 320 of the wearable terminal 100 determines whether the conditions for canceling use assistance control for wired connection are met (S804). If the conditions for canceling use assistance control for wired connection are met, the processor 320 cancels use assistance control for wired connection (S805).

[0147] An example of the determination items related to S804 will be described with reference to Figure 9C. In the example shown in Figure 9C, conditions 1 to 8 relate to conditions 1 to 8 described above with reference to Figure 9A. That is, if conditions 1 to 8 in Figure 9A are no longer satisfied, processor 320 releases the wired connection usage assistance control (conditions 1 to 8).

[0148] Furthermore, the processor 320 releases the use assistance control of the wired connection on the condition that the wired connection is released when the distance between the wearable terminal 100 and the wired device 120 or the connection unit 121 of the device 120 is less than the third distance 205 (Condition 9). In other words, since it is considered that the wired connection has been intentionally released by the user 10, the use assistance control of the wired connection is terminated.

[0149] Furthermore, when the user 10 inputs an instruction to release the control of assistance with wired connection, the processor 320 acquires the input from the user 10 and releases the control of assistance with wired connection (condition 10).

[0150] Returning to Fig. 8 , the description of the processing flow will continue. The user 10 disconnects the wired connection within the third distance 205 (S806). Then, when the user 10 ends use of the wearable terminal 100, the processing ends. On the other hand, when the user 10 continues use of the wearable terminal 100 and makes a new wired connection, the same processing as above is performed (S807).

[0151] In a wearable device 100 such as an HMD, depending on how it is used, the user 10 may be unable to grasp the situation in real space, and if no warning is issued, the user 10 may feel annoyed by the wired connection medium, for example, by being pulled by the connected wired connection medium. Also, the user 10 may feel annoyed by the wired connection medium getting caught on it or by the wired connection medium wrapping around the user's body.

[0152] In contrast, according to the present embodiment, the use assistance control for wired connection is used to notify the user 10 of caution regarding use of the wearable terminal 100, allowing the user 10 to be aware of the situation, thereby enabling better use of the wearable terminal 100. For example, even when a wired connection is established with another peripheral device, the inconvenience caused by the wired connection medium is suppressed, allowing for comfortable and easy operation.

[0153] In the above description, an immersive HMD that is worn on the user's head and displays virtual space, real space, etc. on a display unit has been described as a specific example of the wearable device 100. However, even in the case of an optically transmissive HMD that displays virtual objects on a display unit while visually recognizing physical objects in front of the user, it is clear that when it is determined that the device is more than a predetermined distance away from the location of the device to which it is wired connected, it is possible to notify the user 10 by displaying an icon or message indicating the wired connection status, thereby obtaining the effect of reducing the inconvenience caused by the wired cable 130.

[0154] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0155] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0156] Furthermore, some or all of the above-described configurations, functions, processing units, acquisition sensors, etc. may be implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. The processor may include transistors and other circuits and may be considered circuitry or processing circuitry. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network.

[0157] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0158] The HMD 100 may also be configured by connecting a device such as a smartphone or a tablet. The HMD 100 may be configured to use, for example, the display of the attached smartphone or tablet as the display 308.

[0159] As described above, the first distance 201, the second distance 203, and the third distance 205 may be determined and input by the user 10 and set. Alternatively, for example, the first distance 201, the second distance 203, and the third distance 205 may be automatically calculated by the user 10 inputting the length of the wired cable 130. Here, the length of the wired cable 130 may be input in a format in which the user selects from predetermined lengths, such as 3 m, 5 m, and 7 m, for example. The wearable terminal 100 may then hold data (such as a table) in which the first distance 201, the second distance 203, and the third distance 205 are associated with the length of the wired cable 130, and set these distances by reading out the corresponding first distance 201, second distance 203, and third distance 205 from the input length of the wired cable 130. The first distance 201, the second distance 203, and the third distance 205 are registered in advance in this data as appropriate distances.

[0160] 100 Wearable terminal 101 Left eye gaze sensor 102 Right eye gaze sensor 103 Camera 104 Distance measurement sensor 105 Positioning sensor 106 Microphone 107 Speaker 108 Vibrator 130 Wired cable 304 Acceleration sensor 305 Gyro sensor 306 Geomagnetic sensor 307 Operation input interface 308 Display 320 Processor 330 Memory 345 Wired connection input / output interface 346 Near field wireless communication interface 347 Network communication interface

Claims

1. A wearable device worn by a user, comprising: a display that displays on a display screen a virtual space that may include virtual objects, images of real space and virtual objects, or only images of real space; a wired connection interface to which a wired cable is connected and used for a wired connection with a device; and a processor, wherein the processor identifies the establishment of a wired connection with the device and, when it identifies that the device is located more than a predetermined distance away, issues a notification to the user advising caution regarding use of the wearable device.

2. A wearable terminal as described in claim 1, wherein the processor performs at least one of the following as a notification to warn the user about the use of the wearable terminal: displaying a graphic on the display indicating that a wired connection is in progress; redisplaying the graphic after it has been hidden; and displaying a message indicating that a wired connection is in progress.

3. A wearable terminal as described in claim 1, characterized in that, when the processor identifies that the wearable terminal is further away from the device than a first distance set as the predetermined distance, it displays on the display a wired connection area relating to an area where the wired cable connected to the device may be present, based on the first distance and / or a second distance longer than the first distance, as a notice to warn the user about the use of the wearable terminal.

4. A wearable terminal as described in claim 1, characterized in that the processor executes notifications regarding at least one of the direction toward the device, the location of the device, the distance from the device, and the length of the wired cable as notifications to alert the user to the use of the wearable terminal.

5. A wearable terminal as claimed in claim 1, characterized in that the wearable terminal is equipped with a camera for capturing images of the outside, and the processor displays an image of real space captured by the camera on the display as a notice to warn the user about the use of the wearable terminal.

6. A wearable terminal according to claim 1, characterized in that the wearable terminal is equipped with a camera that captures images of the outside, and the processor generates a virtual object corresponding to the wired cable captured by the camera as a notice to warn users about using the wearable terminal, and displays the generated virtual object on the display.

7. A wearable terminal as claimed in claim 1, characterized in that the wearable terminal is equipped with a camera for capturing images of the outside world and / or a distance sensor for measuring the shape of external objects, and when the processor identifies that the wired cable is straight, it issues a notification indicating that the wired cable is straight as a notification to warn users about the use of the wearable terminal.

8. A wearable terminal as described in claim 1, wherein the processor acquires information regarding the pressure acting on the connection terminal from at least one of a piezoelectric element provided on the connection terminal of the wired cable, a piezoelectric element provided on the connection terminal of the wearable terminal to which the wired cable is connected, and a piezoelectric element provided on the connection terminal of the device to which the wired cable is connected, and when pressure equal to or greater than a predetermined amount is acquired, executes a notification indicating that pressure is being applied to the connection terminal as a notification to warn users of the wearable terminal.

9. A wearable terminal as described in claim 1, wherein the wearable terminal is equipped with a camera that captures images of the outside, a first distance is set as the predetermined distance, and the processor, when it determines that the wearable terminal is at least a third distance away from the device that is shorter than the first distance and that the wired connection has been disconnected, executes at least one of a notification indicating that the wired connection has been disconnected and a video display indicating that the wired connection has been disconnected as a notification to warn users about the use of the wearable terminal.

10. A wearable terminal as described in claim 9, wherein the processor does not issue a notification to warn about use of the wearable terminal when the wearable terminal is located within the third distance from the device and the wired connection is disconnected.

11. An information processing method using a wearable device comprising: a display that displays on a display screen a virtual space that may include virtual objects, images of real space and virtual objects, or only images of real space; a wired connection interface to which a wired cable is connected and used for a wired connection with a device; and a processor, the information processing method comprising the steps of: a step by the processor determining the establishment of a wired connection with the device; a step by the processor acquiring the distance from the device; and a step by the processor issuing a notification to the user of the wearable device to warn them about use when the processor identifies the establishment of a wired connection with the device and identifies that the wearable device is located more than a predetermined distance away from the device.

12. An information processing method according to claim 11, wherein the processor performs at least one of the following as a notification to warn the user about the use of the wearable device: displaying a graphic on the display indicating that a wired connection is in progress; redisplaying the graphic after it has been hidden; and displaying a message indicating that a wired connection is in progress.

13. An information processing method as described in claim 11, characterized in that, when the processor identifies that the wearable terminal is located at a distance from the device that is equal to or greater than a first distance set as the predetermined distance, the processor displays on the display a wired connection area relating to an area where the wired cable connected to the device may be located, based on the first distance and / or a second distance longer than the first distance, as a notification to warn users about the use of the wearable terminal.

14. An information processing method according to claim 11, wherein the processor issues a notification to warn the wearer of at least one of the following: the direction toward the device, the location of the device, the distance from the device, and the length of the wired cable.

15. An information processing method according to claim 11, wherein the wearable device is equipped with a camera that captures images of the outside, and the processor displays an image of real space captured by the camera on the display as a notice to warn the user about the use of the wearable device.

16. An information processing method according to claim 11, wherein the wearable device is equipped with a camera that captures images of the outside, and the processor generates a virtual object corresponding to the wired cable captured by the camera as a notice to warn users about the use of the wearable device, and displays the generated virtual object on the display.

17. An information processing method as described in claim 11, wherein the wearable device is equipped with a camera for capturing images of the outside world and / or a distance sensor for measuring the shape of external objects, and when the processor identifies that the wired cable is straight, it executes a notification indicating that the wired cable is straight as a notification to warn the user about the use of the wearable device.

18. An information processing method as described in claim 11, wherein the processor acquires information regarding the pressure acting on the connection terminal from at least one of a piezoelectric element provided on the connection terminal of the wired cable, a piezoelectric element provided on the connection terminal of the wearable terminal to which the wired cable is connected, and a piezoelectric element provided on the connection terminal of the device to which the wired cable is connected, and when a pressure equal to or greater than a predetermined amount is acquired, executes a notification indicating that pressure is being applied to the connection terminal as a notification to warn users of the wearable terminal.

19. An information processing method as described in claim 11, wherein the wearable device is equipped with a camera that captures images of the outside, a first distance is set as the predetermined distance, and when the processor determines that the wearable device is at least a third distance away from the device that is shorter than the first distance and that the wired connection has been disconnected, it executes at least one of the following notifications to warn the user about the use of the wearable device: a notification indicating that the wired connection has been disconnected; and a video display indicating that the wired connection has been disconnected.

20. An information processing method as described in claim 19, characterized in that the processor does not issue a notification warning about the use of the wearable device when the wearable device is located within the third distance from the device and the wired connection is disconnected.

Images