Head-mounted display, head-mounted display system, and control method

Abstract

The present invention provides a head-mounted display comprising a sensor and a processor. The processor determines whether or not a user is in a vehicle occupancy state on the basis of data acquired from the sensor, and upon determining that the user is in the vehicle occupancy state, issues a notification regarding the configuration of an operation area of the head-mounted display, or alternatively, determines whether or not a shared space exists in the vicinity of the user, and upon determining that a shared space exists in the vicinity of the user, issues a notification regarding the configuration of the operation area of the head-mounted display.

Description

Head-mounted display, head-mounted display system, and control method

[0001] The present invention relates to a head-mounted display, a head-mounted display system, and a control method.

[0002] Currently, immersive head-mounted displays (sometimes referred to as HMDs) are primarily used in residential spaces, but it is conceivable that opportunities for using HMDs will increase even in places where there are many shared areas with others, such as on bullet trains, sleeper trains, and passenger cars, as long as the area is private and safety is ensured.

[0003] Here, Patent Document 1 discloses a technology in which an HMD used by a user communicates with an HMD used by another user nearby, obtains information on relative positions and relative angles, and adjusts the operation spaces so that they do not overlap.

[0004] JP 2012-108577 A

[0005] For example, in a place such as a vehicle where there is generally a self-occupied area and a shared area, the technology of Patent Document 1 makes it possible to set the HMD operation area in the shared area, which means that there is a possibility that the HMD operation area will be set in an area where there is a high possibility of coming into contact with or getting in the way of others.

[0006] Therefore, there is a problem in providing a technique for setting an operation area of ​​an HMD that ensures safety.

[0007] According to one embodiment of the present invention, there is provided a head-mounted display as described below. The head-mounted display includes a sensor and a processor. The processor determines whether the user is in the vehicle based on data acquired from the sensor, and if it determines that the user is in the vehicle, it notifies the user about setting of an operation area. Also provided is a head-mounted display system including the head-mounted display and an in-vehicle system that manages information inside the vehicle. In this system, the head-mounted display and the in-vehicle system are capable of communicating. The head-mounted display communicates with the in-vehicle system to determine whether it is safe for the user to ride, and if it determines that it is safe for the user to ride, it sets an operation area according to the user's riding location, or sets an upper limit of the range in which the operation area can be set at the user's riding location.

[0008] According to one embodiment of the present invention, there is provided the following control method. This control method is performed using a head-mounted display including a sensor and a processor. In this method, the processor determines whether the user is in a vehicle based on data acquired from the sensor. If the processor determines that the user is in a vehicle, it notifies the user regarding setting of an operation area. The processor sets an operation area according to a boarding location designated by the user, or sets an upper limit of a range in which the operation area can be set at a boarding location designated by the user.

[0009] According to the present invention, it is possible to prevent the setting of the operation area of ​​the HMD in an area where there is a high possibility of contact with or disturbance to other people. Note that problems, configurations, and effects other than those described above will become clear from the following description of the embodiment of the invention.

[0010] 13B is a diagram illustrating an example of the hardware configuration of a wearable terminal. FIG. 13C is a diagram illustrating an example of the functional configuration of a wearable terminal. FIG. 13D is a flowchart illustrating an example of an operation area setting operation. FIG. 13E is a flowchart illustrating an example of an operation area setting operation. FIG. 13F is a diagram illustrating an example of a self-occupied area and a shared area. FIG. 13G is a perspective view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. FIG. 13H is a side view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. FIG. 13I is a side view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. FIG. 13I is a perspective view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. FIG. 13I is a perspective view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. FIG. 13I is a perspective view illustrating an example of an upper limit that can be set for the operation area in the self-occupied area. 10 is a flowchart illustrating an example of control for transitioning to a mode in which setting of an operation area is unnecessary or a mode in which setting of an operation area is prohibited. 11 is a flowchart illustrating an example of processing executed by a wearable device and an in-vehicle system in cooperation with each other.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0013] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0014] In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., a memory) and an interface (e.g., a communication port). Therefore, the entity that executes the program and performs the processing may be the processor. Furthermore, the processor may include transistors and other circuits and may be considered as circuitry or processing circuitry. Furthermore, the entity that executes the program and performs the processing may be a controller, device, system, computer, or node that has a processor. The entity that executes the program and performs the processing may be any processing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit is, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), or the like.

[0015] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] First, an example of the hardware configuration of a wearable device will be described with reference to Figure 1. The wearable device 100 can prevent the operation area from being set in an area where there is a high possibility of contact with or disturbance to others, for example, in a vehicle or the like, where a self-occupied area and a shared area exist. In this embodiment, the wearable device 100 is an immersive HMD.

[0017] As shown in FIG. 1, the wearable terminal 100 includes an outer camera 111, an inner camera 112, a distance measurement sensor 113, a position measurement sensor 114, an acceleration sensor 115, a gyro sensor 116, a geomagnetic sensor 117, and the like.

[0018] The wearable terminal 100 includes a sensor. Here, as an example, the sensor may include the above-described components (111 to 117, 133). However, as long as the wearable terminal 100 can execute appropriate processing, the sensor may include, for example, other types of sensors. Furthermore, the sensor may include, for example, the above-described components (111 to 117, 133) with some components omitted.

[0019] The outer camera 111 is a camera that captures images of the surroundings of the wearer of the wearable device 100. For example, while the wearable device 100 is in use, the outer camera 111 captures images in a range from diagonally forward left to diagonally forward right of the wearer. If the wearable device 100 is an immersive HMD, the wearable device 100 can display virtual reality (VR) images. Furthermore, the wearable device 100 can display an image in which a virtual object is superimposed on an external world image using image data acquired by the outer camera 111.

[0020] The in-camera 112 is a camera that captures an image of a person wearing the wearable terminal 100. When the wearable terminal 100 is an HMD, for example, the wearable terminal 100 may track the line of sight of the wearer using image data acquired by the in-camera 112.

[0021] The outer camera 111 and the inner camera 112 acquire image data by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that the parameters (such as the angle of view) of the outer camera 111 and the inner camera 112 are not particularly limited as long as they can perform appropriate processing.

[0022] The ranging sensor 113 is a sensor used to measure the distance and angle to objects around the wearer and acquire distance data and object shape data. For example, a stereo camera may be used. Alternatively, a light detection and ranging (LiDAR) sensor that irradiates an object with laser light such as infrared light and measures the scattered light that bounces back, a time-of-flight (TOF) sensor that measures distance by measuring the reflection time of pulsed light irradiated on the object for each pixel, or a millimeter-wave radar that emits millimeter-wave radio waves and captures the reflected waves may be used.

[0023] An example of the positioning sensor 114 is a GPS (Global Positioning System). The wearable device 100 receives radio waves transmitted from GPS satellites and acquires location information (e.g., latitude and longitude) indicating the current location of the wearable device 100 based on the received radio waves. Alternatively, the outer camera 111 or the ranging sensor 113 may be used as the positioning sensor 114. The current location can be determined by analyzing images captured by the outer camera or three-dimensional point cloud data measured by the ranging sensor. Known technologies for this include Visual Positioning Service (VPS) and Simultaneous Localization and Mapping (SLAM). Alternatively, the positioning sensor 114 may include an acceleration sensor 115, a gyro sensor 116, or a geomagnetic sensor 117. The current position can be determined as a relative position from a reference position, and techniques such as PDR (Pedestrian Dead Reckoning) are known. Positioning may also be performed using wireless communication. The current position can be determined by detecting differences in the strength and arrival time of radio waves from multiple wireless communication access points and performing triangulation. Speed ​​information can also be obtained by obtaining the amount of change in position per specified time.

[0024] The acceleration sensor 115 is a sensor that detects the acceleration of the wearable terminal 100, and the wearable terminal 100 can detect, for example, tilt with respect to the direction of gravity, movement, vibration, and impact based on data acquired by the acceleration sensor 115. The gyro sensor 116 is a sensor that detects the angular velocity of the wearable terminal 100, and the wearable terminal 100 can detect its orientation using the gyro sensor 116. Furthermore, the wearable terminal 100 can detect movement of the head of the user wearing the wearable terminal 100 using the acceleration sensor 115 and gyro sensor 116.

[0025] The geomagnetic sensor 117 is a sensor that detects the Earth's magnetic force and detects the orientation (direction) of the wearable device 100. The wearable device 100 can detect head movement by capturing changes in the geomagnetic field in response to head movement using, for example, a three-axis geomagnetic sensor 117 that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions.

[0026] The vibrator 118 generates vibrations under the control of the processor 135 and converts notification information from the wearable terminal 100 to the user into vibrations. Therefore, the vibrator 118 can transmit notification information from the wearable terminal 100 to the user wearing the wearable terminal 100 as vibrations, thereby improving usability.

[0027] The wearable terminal 100 also includes a communicator 131. The communicator 131 is an interface used for communication, and the wearable terminal 100 connects to a system (in-vehicle system) installed in the vehicle via the communicator 131. The in-vehicle system can manage, for example, information about the inside of the vehicle (such as information about the boarding location in the vehicle) and information about the vehicle (for example, the type of vehicle), and an example of the configuration of this in-vehicle system will be described in detail later.

[0028] The wearable terminal 100 includes a display 132 , a microphone 133 , a speaker 134 , a processor 135 , an operation input interface 136 , a timer 137 , and a memory 138 .

[0029] The display 132 is located in front of both eyes of a user wearing the wearable terminal 100. The processor 135 can display appropriate images, such as VR images, outside world images, and images in which virtual objects are superimposed on outside world images, on the display 132. The user wearing the wearable terminal 100 can view the images displayed on the display 132.

[0030] The microphone 133 is a device that collects and inputs sound, and for example, the user's voice is input to the microphone 133. Furthermore, for example, ambient sounds are input to the microphone 133. The speaker 134 is a device that outputs sound, and for example, outputs sound processed by the wearable device 100.

[0031] The processor 135 is a main body that executes predetermined processing and can be configured using, for example, a CPU (Central Processing Unit). However, as long as it can execute appropriate processing, it may also be configured using, for example, other semiconductor devices.

[0032] The operation input interface 136 may be configured using, for example, a keyboard, key buttons, or a touchpad, and receives operation input from the user. The operation input interface 136 may be located in a position that is easy for the user wearing the wearable terminal 100 to operate, but may also be separated from the main body of the wearable terminal 100 and connected by wire or wirelessly.

[0033] The timer 137 is used, for example, to collect data at regular intervals while the wearable terminal 100 is in use.

[0034] The memory 138 is configured to store programs and data used in predetermined processing, and includes, for example, a main memory device and storage. The main memory device can be a volatile storage medium (e.g., RAM (Random Access Memory)) that allows high-speed reading and writing of information, and is used as a storage area and a working area when the processor 135 processes data. The storage device can be, for example, a non-volatile storage device or non-volatile memory, and can store various data, programs, content, etc.

[0035] The various components (111 to 118, 131 to 138) are connected via buses. In this embodiment, these components (111 to 118, 131 to 138) can input and output various signals and data to and from each other via the buses.

[0036] Next, an example of the functional configuration of the wearable device will be described with reference to Fig. 2. The wearable device 100 includes an image storage unit 211, a distance measurement information storage unit 212, a position / movement speed information storage unit 213, an audio information storage unit 214, a riding safety information storage unit 215, and an HMD operation area information storage unit 216.

[0037] The image storage unit 211 is configured to store appropriate image data, and stores, for example, image data acquired using the outer camera 111 and the inner camera 112. The image storage unit 211 is realized by the memory 138 (more specifically, storage).

[0038] The distance measurement information storage unit 212 stores data on the distance to an object or data on the shape of the object measured using a distance measurement sensor, etc. The distance measurement information storage unit 212 is realized by the memory 138 (more specifically, storage).

[0039] The position / movement speed information storage unit 213 stores position and movement speed data acquired using a positioning sensor, etc. The movement speed information storage unit 213 is realized by the memory 138 (more specifically, storage).

[0040] The voice information storage unit 214 stores voice data acquired using a microphone, etc. The voice information storage unit 214 is realized by the memory 138 (more specifically, storage).

[0041] The ride safety information storage unit 215 stores the acquired ride safety information. The ride safety information will be described in detail later. The ride safety information storage unit 215 is realized by the memory 138 (more specifically, storage).

[0042] The HMD operation area information storage unit 216 stores information about the operation area of ​​the HMD. The HMD operation area information storage unit 216 is realized by the memory 138 (more specifically, storage).

[0043] The wearable terminal 100 also includes a main control unit 231, an image analysis unit 232, a distance measurement information analysis unit 233, a movement speed analysis unit 234, an audio information analysis unit 235, a surrounding situation determination unit 236, a riding safety determination unit 237, and an HMD operation area special setting unit 238. These components (231 to 238) are related to data processing and are realized by a processor 135 and a memory 138 (more specifically, a main storage device). The processor 135 stores appropriate data and programs in the memory 138 (more specifically, a main storage device) and performs data processing.

[0044] The main control unit 231 controls the overall operation of the wearable device 100 according to a predetermined operating program. The image analysis unit 232 performs image analysis. For example, the image analysis unit 232 acquires image data of the external environment captured by the outer camera 111 according to a predetermined program and analyzes the image data. The distance measurement information analysis unit 233 acquires distance data to an object or object shape data acquired by the distance measurement sensor 113 according to a predetermined program and analyzes the distance data or object shape data. The movement speed analysis unit 234 acquires movement speed according to a predetermined program and performs analysis based on the movement speed. The audio information analysis unit 235 acquires audio data collected by the microphone 133 according to a predetermined program and performs analysis based on the audio data. The surrounding situation determination unit 236 determines the surrounding situation according to a predetermined program. For example, it determines whether the user is riding. It also determines, for example, whether a shared space or a self-occupied space exists in the vicinity and the type of these areas (shared space, self-occupied space). The riding safety determination unit 237 determines the user's riding safety according to a predetermined program. The HMD operation area special setting unit 238 performs processing related to setting of the operation area in accordance with a predetermined program.

[0045] Next, an example of the operation of setting the operation area of ​​the wearable device will be described with reference to Fig. 3A. First, the outer camera 111 captures an image of the external environment and acquires the image data. Furthermore, the distance measurement sensor 113 or the like acquires distance data to an object or shape data of the object. Furthermore, the wearable device 100 measures the movement speed. Here, the movement speed may be a value based on, for example, the acceleration sensor 115. Alternatively, it may be a value based on changes in position information acquired by the positioning sensor 113. Furthermore, the microphone 133 collects surrounding sounds and acquires the audio data (S301a).

[0046] Next, the processor 135 of the wearable terminal 100 acquires and analyzes the image data using the image analysis unit 232. In this analysis, the processor 135 performs, for example, image recognition and extracts in-vehicle items and the like from the image. In-vehicle items may include, for example, vehicle seats, vehicle berths, seat belts, and the like.

[0047] The processor 135 also acquires and analyzes the distance data or the shape data in the distance measurement information analysis unit 233. In this analysis, the processor 135 determines, for example, whether the distance to the object being measured is equal to or shorter than a predetermined distance. Here, the predetermined distance can be roughly defined as the sum of the gap between the front and rear seats in the longitudinal direction of a typical passenger vehicle (such as a bullet train or bus) and the seat depth (the shortest horizontal distance from the front edge of the center of the seat to the rear edge (or the rear edge of the backrest if there is one)). The processor 135 also determines whether the acquired object shape data matches a predetermined shape pattern. Here, the predetermined shape pattern can include, for example, the shape patterns of a vehicle seat, a vehicle berth, a seatbelt, etc.

[0048] The processor 135 also acquires and analyzes the movement speed in the movement speed analysis unit 234. In this analysis, the processor 135 determines, for example, whether the movement speed of the wearer is equal to or greater than a predetermined value. Here, the predetermined value may be, for example, a value corresponding to a speed that is considered to be running, rather than slowing down. The predetermined value may be, for example, 10 km / h (absolute speed).

[0049] The processor 135 also acquires and analyzes the voice data in the voice information analysis unit 235. In this analysis, the processor 135 determines, for example, whether the collected and input voice matches a predetermined voice pattern. Here, the predetermined voice pattern may include, for example, an announcement, a voice pattern of a navigation system, etc.

[0050] Then, the processor 135 determines, in the surrounding situation determination unit 236, whether the wearer (user) of the wearable terminal 100 is in a vehicle (S302a).

[0051] For example, when an in-vehicle object is extracted by the image analysis unit 232, the processor 135 may use the surrounding situation determination unit 236 to recognize that the user is inside a vehicle (such as a bullet train, a passenger car, a sleeper train, or a bus) and determine that the user is in the vehicle. The processor 135 may also use information such as the arrangement pattern of the in-vehicle objects and the number of the in-vehicle objects to determine whether the user is in the vehicle. That is, for example, when the image analysis unit 232 recognizes a predetermined arrangement pattern of vehicle seats, a predetermined number or more of vehicle seats, etc., the processor 135 may use the surrounding situation determination unit 236 to recognize that the user is inside a vehicle (such as a bullet train, a passenger car, a sleeper train, or a bus) and determine that the user is in the vehicle.

[0052] In addition, if the distance measurement information analysis unit 233 determines that the distance to the object being measured is less than a predetermined distance, or if the acquired shape data of the object is determined to match a predetermined shape pattern, the processor 135 may use the surrounding situation judgment unit 236 to recognize that the user is inside a vehicle (such as a Shinkansen, passenger car, sleeper train, or bus) and determine that the user is inside the vehicle.

[0053] Furthermore, if the movement speed analysis unit 234 determines that the movement speed of the wearer is equal to or greater than a predetermined value, the processor 135 may use the surrounding situation determination unit 236 to recognize that the wearer is inside a vehicle (such as a Shinkansen, a passenger car, a sleeper train, or a bus) and determine that the user is in a vehicle.

[0054] Furthermore, if the audio information analysis unit 235 determines that the collected and input audio matches a predetermined pattern of audio, the processor 135 may use the surrounding situation determination unit 236 to recognize that the user is inside a vehicle (such as a Shinkansen, passenger car, sleeper train, or bus) and determine that the user is inside the vehicle.

[0055] The processor 135 may determine, in the surrounding situation determination unit 236, that the user is in the vehicle based on multiple results from the image analysis unit 232, the distance measurement information analysis unit 233, the movement speed analysis unit 234, and the audio information analysis unit 235. For example, when the image analysis unit 232 extracts an item inside the vehicle and the movement speed analysis unit 234 determines that the movement speed of the wearer is equal to or greater than a predetermined value, the processor 135 may determine, in the surrounding situation determination unit 236, that the user is in the vehicle.

[0056] If it is determined in S302a that the user is not in a riding state, the process of S301a is performed (S302a-NO). On the other hand, if it is determined that the user is in a riding state (S302a-YES), the process of S303a is performed. In S303a, the processor 135 notifies the user, via the HMD operation area special setting unit 238, that special setting of the HMD operation area is required. Here, special setting of the HMD operation area means setting of the operation area only in the self-occupied area that is unlikely to come into contact with or disturb others.

[0057] In S304a, when specially setting the HMD operation area, the processor 135 acquires image data of the interior of the vehicle captured using the outer camera 111 and distance data or shape data acquired using the distance sensor 133 or the like in the HMD operation area special setting unit 238. Then, as shown in Fig. 4, the processor 135 recognizes the space between the front and rear seats as a passenger area from the captured image or the distance measurement data or shape data, and classifies the space into a passenger area and an aisle area to identify self-occupied places, shared places, and these areas (self-occupied areas, shared areas). Note that the processor 135 may acquire and process a distance image based on the distance measurement data.

[0058] In the example of Figure 4, a black circle A1 indicates a boarding location area (i.e., a self-occupied area), and a white circle A2 indicates an aisle location area (i.e., a shared area). A self-occupied area is an area of ​​a location that a user occupies and can set an operation area, such as an area of ​​a passenger boarding location in a vehicle. A shared area is an area of ​​a location that is shared by passengers in a vehicle, such as an aisle location in a vehicle.

[0059] In S305a, the user specifies the boarding location. Then, in S306a, the user sets the operation area within the upper limit of the self-occupied area according to the boarding location. Here, as shown in Figures 5 and 6, for example, the space extending from the floor or seat to the ceiling (range A3 indicated by the imaginary line in the figures) is set as the upper limit of the self-occupied area so as not to overlap with the seat in front. The operation area of ​​the HMD can be set within this space.

[0060] In S306a, the wearable terminal 100 accepts an operation by the user, and the processor 135 sets an operation area in accordance with the user's operation using the HMD operation area special setting unit 238. Note that instead of the user setting the operation area, the processor 135 may automatically set the self-occupied area as the operation area using the HMD operation area special setting unit 238. The wearable terminal 100 may be configured so that the method for setting the operation area can be selected by the user's operation. Furthermore, for example, in a case where seats are arranged in a row, the processor 135 may perform processing using the HMD operation area special setting unit 238 so as to exclude an area where an armrest is located in the middle from the self-occupied area.

[0061] However, the environment of the boarding location may change. Therefore, even after the operation area is set in S306a, the processor 135 acquires image data, distance data, and shape data in the HMD operation area special setting unit 238, and monitors whether the environment of the user's boarding location has changed (S307a, S308a).

[0062] 7, an example will be described in which the processor 135 determines in S308a that the environment of the boarding location has changed. If the operation area is set and the user then reclines the seatback of the seat in which they are sitting, the upper limit of the self-occupied area will become larger, as shown by range A4 indicated by the imaginary line in Fig. 7. Therefore, the processor 135 recognizes the change in the environment of the boarding location using the image data, distance data, and shape data acquired in S307a in the HMD operation area special setting unit 238, and notifies the user of the resetting of the operation area (S309a).

[0063] Then, in S310a, the user resets the operation area within the upper limit of the self-occupied area. The wearable terminal 100 accepts an operation by the user, and the processor 135 resets the operation area in accordance with the user's operation using the special HMD operation area setting unit 238. Note that instead of the user setting the operation area, the processor 135 may automatically set the self-occupied area as the operation area using the special HMD operation area setting unit 238. The wearable terminal 100 may be configured so that the user can select which method to use to set the operation area. If the operation area has been reset, the process returns to S304a (YES in S310a). If the operation area is not to be reset, the process ends (NO in S310a).

[0064] In the example of Figure 7, a case is described in which the user reclines the backrest of the seat in which they are sitting, and the upper limit range of the self-occupied area becomes larger than the range in Figure 6. However, for example, if the backrest of the seat in front of them is reclined and the upper limit range of the self-occupied area becomes smaller, the wearable terminal 100 will also notify the user about resetting the operation area.

[0065] As also explained in the example of FIG. 4, the operation shown in FIG. 3B may be performed as an example of the operation of setting the operation area of ​​the wearable device. In this case, the question "Is the user in a vehicle?" (S302a) in FIG. 3A is determined as "Is a shared place present around the user?" (S302b) by analyzing information obtained by at least one of imaging, distance measurement, movement speed measurement, and sound collection. For example, if one of the conditions for determining that the user is in a vehicle, as described above, is satisfied, it may be determined that a shared place exists around the user. Furthermore, the "boarding location" in FIG. 3A is determined as a "self-occupied location."

[0066] Next, an example of setting the operation area in another environment in FIG. 3A will be described with reference to FIGS. 8A and 8B. As mentioned above, this may also be applied to the processing of the operation shown in FIG. 3B. Similar descriptions may be omitted as appropriate. In S304a, the processor 135, in the HMD operation area special setting unit 238, recognizes the space above the bed as a riding area based on the captured image or the distance measurement data and shape data, and classifies the riding area from other areas to identify the self-occupied place, shared place, and these areas (self-occupied area, shared area). The processor 135 may also acquire and process a distance image based on the distance measurement data.

[0067] The user specifies a boarding location (S305a), and similarly to the above, an operation area is set within the range of the upper limit of the self-occupied area corresponding to the boarding location (S306a). Here, as will be described later, the upper limit of the self-occupied area is adjusted, and as a result, as shown in Fig. 8A, for example, the space above and to the side of the bed (range A5 indicated by the imaginary line in the figure) becomes the upper limit of the self-occupied area. The operation area of ​​the HMD can be set within this space.

[0068] Thereafter, the processor 135 acquires image data, distance data, and shape data in the HMD operation area special setting unit 238, monitors whether the environment of the user's boarding location has changed (S307a, S308a), and determines that the environment of the boarding location has changed when the bed curtain is closed as shown in Fig. 8B . Then, the processor 135 notifies the user about the resetting of the operation area in the HMD operation area special setting unit 238 (S309a).

[0069] Then, in S310a, if the operation area is reset within the upper limit of the current self-occupied area (i.e., range A6 shown by the virtual line in Figure 8B), processing returns to S304a (S310a-YES), and if the operation area is not reset, processing ends (S310a-NO).

[0070] Although an example has been described in which the curtains are closed and the upper limit of the self-occupied area becomes smaller, the wearable terminal 100 also notifies the user about resetting the operation area in the same way when, for example, the curtains are opened and the upper limit of the self-occupied area becomes larger.

[0071] Next, an example in which the wearable device adjusts the range of the upper limit of the self-occupied area will be described with reference to Fig. 9A. As shown in Fig. 9A, when the processor 135 does not detect an object in a predetermined space on the side of the boarding location (in this example, a predetermined space on the side of the berth) based on the acquired image data, distance data, and shape data, the HMD operation area special setting unit 238 may expand this predetermined space as the self-occupied area. That is, as indicated by the two-way arrow AS1 in Fig. 9A, the range of the upper limit of the self-occupied area may be adjusted.

[0072] Here, the size of the predetermined space may be determined, for example, according to the type of in-vehicle item and may be registered in advance in memory 138 (more specifically, storage). For example, it is considered that the passage of passengers and the like is low in the aisles beside the berths. Therefore, in the case of a berth, the size of the predetermined space may be set to a certain size and registered. On the other hand, it is considered that the passage of passengers and the like is high in the aisles beside the seats. Therefore, in the case of a seat, the size of the predetermined space may be set to 0 and registered. In other words, in the case of a seat, a value that does not expand the upper limit of the self-occupied area may be registered.

[0073] The processor 135 recognizes an item inside the vehicle at the boarding location using, for example, image data, distance data, and shape data, and reads data of a predetermined space corresponding to the item inside the vehicle from the memory 138 (more specifically, storage). Then, if the processor 135 does not detect an object in the predetermined space corresponding to the item inside the vehicle based on the image data, distance data, and shape data, it may perform processing to expand the predetermined space corresponding to the item inside the vehicle as an upper limit range of the self-occupied area.

[0074] The processor 135 may also adjust the range of the upper limit of the self-occupied area based on the following method: That is, the processor 135 may adjust the range of the upper limit of the self-occupied area, for example, in the HMD operation area special setting unit 238, as shown in FIG.

[0075] In FIG. 9B , the user is using the wearable device 100 while it protrudes into a predetermined space on the side of the boarding location. Here, the processor 135 analyzes the user's body size from captured images or distance measurement data and shape data. The processor 135 may then adjust (expand or contract) the size of the predetermined space according to the user's body size, and adjust (expand or contract) the upper limit of the self-occupied area. That is, as indicated by the bidirectional arrow AS3 in the figure, the upper limit of the self-occupied area may be adjusted.

[0076] Furthermore, the processor 135 can set the upper limit of the self-occupied area in the vertical direction using the special HMD operation area setting unit 238 so that operation is not difficult. On the other hand, the processor 135 may adjust the upper limit of the self-occupied area in the vertical direction, for example. That is, the upper limit of the self-occupied area may be adjusted as shown by the two-way arrow AS2 in FIG. 9A and the two-way arrow AS4 in FIG. 9B.

[0077] The processor 135 may adjust the upper limit of the self-occupied area to a predetermined height from the seat where the user sits, for example, using a captured image, distance measurement data, or shape data. Here, the predetermined height is registered in advance in the memory 138 (more specifically, storage), and may be set to, for example, a value approximately equal to the vertical height of the seat back of a typical vehicle (such as a bullet train or bus).

[0078] On the other hand, the processor 135 may recognize the height of the user's shoulders using, for example, a captured image or distance measurement data or shape data, and adjust the upper limit of the self-occupied area so that the shoulder height becomes the upper limit of the range of the self-occupied area.

[0079] In this way, it is possible to expand the upper limit of the self-occupied area while preventing the operation area from being set in an area where there is a high possibility of contact with or disturbing others in the vehicle. Furthermore, by adjusting the upper limit of the self-occupied area in the vertical direction, it is possible to prevent the user from using the wearable device 100 while standing in the vehicle, for example, thereby improving the safety of the user.

[0080] For example, if the vehicle is equipped with multiple berths and a user is using an upper berth, there is a risk of the user falling if the operation range extends to the side of the berth. Therefore, the processor 135 may use the captured image or the distance measurement data and shape data to detect when the user's feet touch the floor, and then perform processing to extend the upper limit of the self-occupied area to the side of the boarding location.

[0081] Next, an example of the configuration of an in-vehicle system will be described with reference to Fig. 10. As shown in Fig. 10, the in-vehicle system 500 is capable of communicating with the wearable device 100 and includes a processor 511, a memory 512, a communicator 513, a seat 514, and a seat belt 515.

[0082] The processor 511 is a main body that executes predetermined processing and can be configured using, for example, a CPU (Central Processing Unit). However, as long as it can execute appropriate processing, it may also be configured using, for example, other semiconductor devices.

[0083] The memory 512 is configured to store programs and data used for predetermined processing, and includes, for example, a main memory device and storage. The main memory device can be a volatile storage medium (e.g., RAM (Random Access Memory)) that allows high-speed reading and writing of information, and is used as a storage area and a working area when the processor 511 processes data. The storage is, for example, a non-volatile storage or non-volatile memory, and stores various data, programs, etc.

[0084] The communicator 513 is an interface device used for communication, and the in-vehicle system 500 connects to the wearable terminal 100 and the like via the communicator 513 .

[0085] The processor 511, memory 512, and communication device 513 are connected via a bus. In this embodiment, these components (511 to 513) can input and output various signals and data to and from each other via the bus.

[0086] The processor 511 is also connected via a bus to a component that detects the use status of the seat 514, and in this embodiment, various signals and data can be input and output via the bus. Here, examples of the component that detects the use status of the seat 514 include a sensor (load sensor, pressure sensor, etc.) provided on the seat 514, an in-vehicle camera that captures an image of the seat 514, etc.

[0087] The processor 511 can determine whether a seat in the vehicle is occupied (in use) based on the weight detected by a load sensor, a pressure sensor, etc. provided in each of the vehicle seats 514. The processor 511 can also determine which seat 514 in the vehicle a passenger is sitting in by performing known image authentication on an image acquired by an in-vehicle camera. The processor 511 may make the determination using both the detection results of sensors such as the load sensor and pressure sensor and the image authentication results from the in-vehicle camera.

[0088] The processor 511 is also connected via a bus to a component that detects the fastening / unfastening status of the seat belt 515, and in this embodiment, various signals and data can be input and output via the bus. Examples of the component that detects the fastening / unfastening status of the seat belt 515 include a switch that detects the fastening / unfastening of the seat belt 515 in response to insertion and removal of the tongue plate from the buckle, a known seat belt sensor, an in-vehicle camera, etc.

[0089] The processor 511 can determine whether or not a seat belt 515 in the vehicle is being used based on the detection results of switches and sensors provided for each seat belt 515 in the vehicle. The processor 511 can also determine whether or not a seat belt 515 in the vehicle is being used by performing known image authentication on an image acquired by an in-vehicle camera. The processor 511 may make the determination using both the detection results of the switches, seat belt sensors, etc. and the image authentication results from the in-vehicle camera.

[0090] The vehicle in which the in-vehicle system 500 is installed is not particularly limited, and the in-vehicle system 500 can be installed in a bullet train, a passenger car, a sleeper train, a bus, or the like. The configuration of the in-vehicle system 500 can be changed as appropriate. For example, in a vehicle in which a berth is installed at the boarding location, the in-vehicle system 500 may be connected to a configuration for detecting the berth usage status via a bus. Here, the configuration for detecting the berth usage status can be, for example, the same as that for the seat 514. Furthermore, in a vehicle without seats 514 (e.g., a vehicle equipped with only berths), the in-vehicle system 500 can be configured such that inputs and outputs related to the seats 514 are omitted and that inputs and outputs related to the berths are included. Furthermore, in a vehicle without seat belts 515, the in-vehicle system 500 can be configured such that inputs and outputs related to the seat belts 515 are omitted.

[0091] Next, an example of the functional configuration of the in-vehicle system will be described with reference to Fig. 11 . The in-vehicle system 500 includes a riding safety information storage unit 611. The riding safety information storage unit 611 stores riding safety information acquired by the processor 511. Here, the riding safety information includes, for example, information on the occupancy (use) of seats 514 in the vehicle and information on the wearing status of seat belts 515. The riding safety information storage unit 611 is realized by a memory 512 (more specifically, a storage).

[0092] The in-vehicle system 500 links the riding safety information to the ID (e.g., seat number) of each seat 514 in the vehicle and manages it for each seat 514. Therefore, one seat 514 is linked to one piece of riding safety information corresponding to this seat 514. Note that, for example, if there is no seat belt 515 in the seat 514, information indicating that there is no seat belt 515 may be linked to the seat 514.

[0093] The riding safety information can be changed as appropriate. For example, in a vehicle without seat belts 515, information on the wearing status of the seat belts 515 may be omitted from the riding safety information. In a vehicle with berths, the in-vehicle system 500 may link the riding safety information to the ID of each berth in the vehicle and manage it for each berth. The riding safety information related to berths includes, for example, information on the occupancy (use) of the berth in the vehicle.

[0094] The in-vehicle system 500 also includes a main control unit 631 and a riding safety information acquisition unit 632. These components (631, 632) are related to data processing and are realized by a processor 511 and a memory 512 (more specifically, a main storage device). The processor 511 stores appropriate data and programs in the memory 512 (more specifically, a main storage device) and performs data processing.

[0095] The main control unit 631 controls the overall operation of the in-vehicle system 500 in accordance with a predetermined operation program. The ride safety information acquisition unit 632 acquires ride safety information for each boarding location using data acquired by a configuration for detecting the status of the boarding location (seat 514, berth, etc.) and a configuration for detecting the fastening and unfastening status of the seat belt 515, if present, and stores the acquired ride safety information in the ride safety information storage unit 611.

[0096] Next, an example of processing executed by the wearable device and the in-vehicle system in cooperation with each other will be described with reference to FIG. 12 . Note that explanations similar to those described above may be omitted. Here, an example will be described in which the boarding location is a seat. However, if the boarding location is a berth, for example, the seat is replaced with a berth and similar processing is executed. The processor 135 of the wearable device 100 can execute this processing using the boarding safety determination unit 237, the HMD operation area special setting unit 238, etc.

[0097] The wearable terminal 100 (more specifically, the processor 135) performs the same processes as in S301a to S304a described above (S1201 to S1204), and the user specifies the boarding location (S1205) in the same manner as in S305a described above.

[0098] Next, the processor 135 performs the following steps S1206a to S1206e in the riding safety determination unit 237. That is, the processor 135 requests riding safety information for the seat 514 at the boarding location from the vehicle system 500 (S1206a). In response to this request, the in-vehicle system 500 requests seat information from the wearable device 100 (S1206b).

[0099] Then, the user inputs the seat number of the seat 514 at the boarding location into the wearable terminal 100, and the processor 135 of the wearable terminal 100 transmits this seat number to the in-car system 500 as seat information (S1206c).

[0100] The in-vehicle system 500 acquires the riding safety information corresponding to the seat number acquired in S1206b to S1206c from the riding safety information storage unit 611 and transmits the acquired riding safety information to the wearable device 100 (S1206d). Then, the wearable device 100 acquires the riding safety information of the boarding location (S1206e).

[0101] After the wearable device 100 acquires the riding safety information, the processor 135, in the riding safety determination unit 237, determines whether the user's riding is safe (S1207). If the wearable device 100 acquires riding safety information indicating that the user is seated in the seat 514 at the boarding location and is fastening the seat belt 515, if one is available, the processor 135 determines that the user is in a safe state, and the process proceeds to S1208 (S1207-YES). That is, if the user is seated in the seat 514 at the boarding location and is fastening the seat belt 515, if one is available for the seat 514, when the wearable device 100 transmits the boarding number to the in-car system 500 in S1206c, the process proceeds to S1208. On the other hand, if the wearable terminal 100 has acquired riding safety information indicating that this is not the case, it is determined that the user's riding state is unsafe, and the process returns to S1206a (S1207-NO).

[0102] When processing proceeds to S1208, processing similar to S306a to S310a described above is performed (S1208 to S1212). Alternatively, information regarding the HMD operation area may be stored in memory 512 of in-vehicle system 500, and processor 511 may transmit the information regarding the HMD operation area along with riding safety information to wearable terminal 100. Based on the received information, wearable terminal 100 may determine whether the user is in a safe state and, if the user is in a safe state, perform special settings for the HMD operation area. Alternatively, if the riding safety information transmitted by processor 511 to wearable terminal 100 indicates that the user is not in a safe state, processor 511 may transmit only the riding safety information to wearable terminal 100. When the operation area is reset in S1212, processing returns to S1204 (YES in S1212).

[0103] 13A shows an example of a seat with a seat belt. Even in the case of a seat 514 with a seat belt 515, as shown in FIG. 13B, the upper limit of the self-occupied area is the space extending from the floor or seat 514 to the ceiling (area A7 shown by the imaginary line in the figure) so as not to overlap with the areas of other seats in front and beside the seat. Note that if the seats are connected together as shown in FIGS. 13A and 13B and have armrests in the middle, the processor 135 may perform processing in the HMD operation area special setting unit 238 so as not to include the area where the armrests are located in the self-occupied area.

[0104] Next, with reference to Figure 14, an example of checking the user's riding safety using the camera of the wearable device and setting the operation space will be described. The processor 135 of the wearable device 100 can execute processing using the riding safety determination unit 237, the HMD operation area special setting unit 238, etc. This example differs from the processing in Figure 12 in that the riding safety determination unit 237 determines riding safety without performing communication. Note that explanations similar to those described above may be omitted. Also, here, an example will be described in which the riding location is a seat, but if the riding location is a berth, for example, the seat will be replaced with a berth and similar processing will be performed.

[0105] The wearable terminal 100 (more specifically, the processor 135) performs the same processes as those in S1201 to S1204 described above (S1401 to S1404), and the user specifies the boarding location (S1405) in the same manner as in S1205 described above.

[0106] Next, the processor 135 performs the following step S1406 in the riding safety determination unit 237. That is, the processor 135 acquires image data captured using the outer camera 111, and distance data and shape data acquired using a distance measurement sensor or the like (S1406).

[0107] The processor 135 then determines whether the user's ride is safe using the ride safety determination unit 237 (S1407). Here, the processor 135 may determine, for example, by performing well-known image authentication, whether the user is seated in the seat 514 and, if a seat belt 515 is present, whether the seat belt 515 is fastened. The processor 135 may also process the distance data as a distance image, for example. If the processor 135 determines that the user's ride is safe (i.e., if the processor 135 determines that the user is seated in the seat 514 and, if a seat belt 515 is present, whether the seat belt 515 is fastened), the process proceeds to S1408 (YES in S1407). On the other hand, if the processor 135 determines that the user's ride is not safe, the process returns to S1406 (NO in S1407).

[0108] If the process proceeds to S1408, the same processes as those of S1208 to S1212 described above are performed (S1408 to S1412). If the operation area is to be reset in S1412, the process returns to S1404 (S1412-YES).

[0109] As described above (i.e., as explained using Figures 12 and 14), by checking the safety of the user's ride, the wearable terminal 100 can be used in a state where the user's safety is increased.

[0110] 15, for example, when the user is wearing a seat belt 515, the user's safety is generally ensured, and it may be considered that the wearable device 100 does not need to set an operation area. Next, an example of control for transitioning to a mode in which an operation area is not set will be described with reference to FIG. 16. Note that descriptions similar to those described above may be omitted.

[0111] The wearable terminal 100 (more specifically, the processor 135) performs the same processes as those in S1401 to S1404 described above (S1601 to S1604), and the user specifies the boarding location (S1605) in the same manner as in S1405 described above.

[0112] The processor 135, in the riding safety determination unit 237, performs the same processing as in S1406 to S1407 described above to determine whether the user's riding is safe (S1606 to S1607). That is, in S1606, the processor 135 acquires image data captured using the outer camera 111, and distance data and shape data acquired using a distance measurement sensor or the like. If the processor 135 determines in S1607 that the user's riding is safe, the process proceeds to S1608 (S1607-YES). If not, the process returns to S1606 (S1607-NO). In this example, the processor 135 determines the user's riding safety based on image data, distance data, and shape data in the riding safety determination unit 237. However, the processor 135 may also determine the user's riding safety based on other methods (for example, a method of communicating with the in-vehicle system 500 as shown in S1206a to S1206e and S1207 of FIG. 12) in the riding safety determination unit 237.

[0113] If it is confirmed in S1607 that it is safe to ride, the wearable terminal 100 transitions to a mode in which setting of the operation area is not required by the HMD operation area special setting unit 238. The operation area setting not required mode (HMD operation area setting not required mode in the figure) is a mode in which it is not necessary to set an upper limit range of the operation area and to set the operation area, and in this mode, the user can use the wearable terminal 100 without setting the operation area. In other words, in this mode, the use of the wearable terminal 100, which involves user movement, is permitted without any restrictions imposed by the operation area.

[0114] Furthermore, if the user is not riding in a specific vehicle (a vehicle exempt from wearing a seat belt) and there is no seat belt at the boarding location, the operation area setting prohibition mode may be entered. Next, with reference to FIG. 17 , an example of control for entering the operation area setting prohibition mode (HMD operation area setting prohibition mode) will be described. Here, the operation area setting prohibition mode is a mode in which the setting of an operation area is prohibited, so that the wearable terminal 100 can be used without user movement, but use of the wearable terminal 100 with user movement is prohibited. Note that explanations similar to those described above may be omitted.

[0115] The wearable terminal 100 (more specifically, the processor 135) performs the same processes as those in S1601 to S1604 described above (S1701 to S1704), and the user specifies the boarding location (S1705) in the same manner as in S1605 described above.

[0116] Then, the processor 135 determines whether the user is riding in a specific vehicle using the riding safety determination unit 237 (S1711). Here, the specific vehicle is a vehicle that does not have seat belts installed at the boarding area, such as a Shinkansen.

[0117] The processor 135 can determine whether the vehicle is a specific vehicle based on an appropriate method. For example, the user may input information about the vehicle type into the wearable terminal 100, which causes the processor 135 to determine that the vehicle is a specific vehicle. Alternatively, the wearable terminal 100 may acquire information about the vehicle type via communication and make the determination. Alternatively, the processor 135 may perform image recognition using captured image data to determine whether the vehicle is a specific vehicle.

[0118] If the processor 135 determines in S1711 that the vehicle is not a specific vehicle, the process proceeds to S1712 (S1711-NO). On the other hand, if the processor 135 determines that the vehicle is a specific vehicle (S1711-YES), the process proceeds to S1706.

[0119] The processor 135 determines in the boarding safety determination unit 237 whether or not a seat belt is present at the boarding location specified by the user in S1705 (S1712).

[0120] The processor 135 can determine whether a seat belt is fastened based on an appropriate method. For example, the user may input information about the presence or absence of a seat belt into the wearable device 100, and the processor 135 may determine whether a seat belt is fastened. Alternatively, the user may input information identifying a seat at the boarding location (e.g., a seat number) into the wearable device 100, and the wearable device 100 may communicate with the vehicle system 500 and determine whether a seat belt is fastened based on the information acquired from the vehicle system 500. Alternatively, the wearable device 100 may acquire information identifying a seat at the boarding location through image recognition using image data, communicate with the vehicle system 500, and determine whether a seat belt is fastened based on the information acquired from the vehicle system 500. Alternatively, the wearable device 100 may determine whether a seat belt is fastened in a seat at the boarding location by performing image recognition using image data of the seat at the boarding location.

[0121] If the processor 135 determines in S1712 that the seat belt is not fastened (NO in S1712), the processor 135 performs control to transition to a mode in which the setting of the operation area is prohibited without confirming the safety of the vehicle (S1713). On the other hand, if the processor determines that the seat belt is fastened (YES in S1712), the processor 135 performs a safety check similar to S1606 to S1607 (S1706 to S1707). If the safety of the vehicle is confirmed (i.e., the user is seated and has the seat belt fastened) (YES in S1707), the processor 135 performs control to transition to a mode in which the setting of the operation area is not required (S1708). The processor 135 may perform the safety check based on image data, distance data, and shape data, or may determine the safety of the user based on other methods (e.g., a method of communicating with the in-vehicle system 500).

[0122] When the user's safety is confirmed in a vehicle, setting of the operation area of ​​the wearable terminal 100 is not required, allowing the user to easily use the wearable terminal 100. Furthermore, it is possible to prevent the user from using the wearable terminal 100 when the user's safety is not confirmed in the vehicle. On the other hand, when it is confirmed that the user is not riding in a specific vehicle and that there is no seat belt at the riding location, the wearable terminal 100 can be used without user movement, but contact is prevented by transitioning to a mode that prohibits use of the wearable terminal 100 that involves user movement. In step S1713 of the above process, the processor 135 may perform control to transition to a use prohibition mode (HMD use prohibition mode) that prohibits use of the wearable terminal 100.

[0123] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, for example, other configurations may be added, deleted, or replaced with part of the configuration of the embodiment.

[0124] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0125] The wearable terminal 100 may acquire the riding safety information by communicating with the in-vehicle system 500, for example, as follows: That is, as shown in Fig. 18, the wearable terminal 100 performs the processes of S1806a to S1806d instead of the processes of S1206a to S1206e.

[0126] In S1806a, the wearable terminal 100 requests the in-vehicle system 500 for riding safety information, and also transmits the position information of the wearable terminal 100 acquired using the positioning sensor 114 to the in-vehicle system 500.

[0127] In S1806b, the in-vehicle system 500 receives the request for ride safety information and also receives the location information of the wearable terminal 100. Then, the in-vehicle system 500 acquires the ride safety information for the corresponding ride location (seat, berth, etc.).

[0128] Here, a GPS sensor is installed at each boarding location in the vehicle, and the in-vehicle system 500 may acquire location information of the boarding location using the GPS sensor at the boarding location, and manage the location information of each boarding location by linking it to the ID of each boarding location.

[0129] The in-vehicle system 500 may then compare the location information received from the wearable terminal 100 with the location information of the boarding location that it manages, and obtain the boarding safety information linked to the ID of the boarding location with the smallest difference in location information.

[0130] Furthermore, a GPS sensor may be provided at a specific location in the vehicle (for example, in the driver's cab), and the in-vehicle system 500 may manage the position information of each boarding location based on the location where the GPS sensor is provided. That is, the in-vehicle system 500 may manage the position information of each boarding location in the vehicle (i.e., n boarding locations) based on the position coordinates (X ｎ , Y ｎ ) the location information of each boarding location may be managed by linking it to the ID of each boarding location.

[0131] The in-vehicle system 500 then receives the position information of the wearable device 100 and calculates the position coordinates of the wearable device 100 using the received position information, with the position coordinates of the GPS sensor set as the reference (0, 0). The in-vehicle system 500 then identifies the boarding location whose position coordinates are closest to the calculated position coordinates of the wearable device 100 (i.e., identifies the boarding location whose distance from the wearable device 100 is the shortest), and may acquire boarding safety information linked to the ID of the identified boarding location.

[0132] In S1806c, the in-vehicle system 500 transmits the riding safety information acquired in S1806b to the wearable terminal 100. Then, in S1806d, the wearable terminal 100 receives the riding safety information.

[0133] As described with reference to FIG. 12 , when a user inputs boarding location information (e.g., seat information), it is possible that the user may input the information incorrectly. To prevent this, when the boarding location information is input and transmitted to the vehicle, the wearable device 100 may transmit the acquired location information and / or captured image information together. The in-vehicle system 500 may then perform processing using the received location information and / or captured image information.

[0134] When using location information, as described above, a GPS sensor may be provided in the vehicle, and the in-vehicle system 500 may manage the location information of each boarding location by linking it to the ID of each boarding location. The in-vehicle system 500 then identifies the boarding location using the location information received from the wearable device 100 in the same manner as described above, and confirms whether the identified boarding location number matches the boarding location number entered by the user. If the in-vehicle system 500 confirms that the boarding location numbers match, it transmits boarding safety information for the boarding location corresponding to the boarding location number entered by the user to the wearable device 100. If the boarding location number does not match, it transmits a notification to the wearable device 100 prompting the user to re-enter the boarding location number. When the boarding location number is re-entered and transmitted to the in-vehicle system 500, the in-vehicle system 500 confirms the boarding location number and performs the same process.

[0135] When using information from a captured image, the in-vehicle system 500 identifies the boarding location number by, for example, extracting the boarding location number from the image. Then, the in-vehicle system 500 confirms that the identified boarding location number matches the boarding location number entered by the user. Then, as described above, if the in-vehicle system 500 confirms that the boarding location numbers match, it transmits boarding safety information for the boarding location corresponding to the boarding location number entered by the user, and if it cannot confirm that the boarding location numbers match, it prompts the user to re-enter the boarding location number. If the boarding location number is re-entered, the in-vehicle system 500 performs the same processing.

[0136] Furthermore, the in-vehicle system 500 may manage reservation information for the user's boarding location by linking it to the ID of the boarding location. When requesting input of the boarding location number, the in-vehicle system 500 may also request input of the boarding location reservation information. The in-vehicle system 500 may confirm a match between the reservation information, and if the match between the reservation information is confirmed, may transmit boarding safety information for the boarding location number entered by the user, and if the match between the reservation information is not confirmed, may send a notification prompting the user to re-enter the boarding location number.

[0137] A communication device used for communication with the vehicle system 500 may be provided at each boarding location. The vehicle system 500 may manage the communication device at each boarding location by linking it to the ID of the boarding location. By connecting to the communication device at the boarding location, the wearable terminal 100 can communicate with the in-vehicle system 500 via this communication device. In communication with the wearable terminal 100, the in-vehicle system 500 may identify the boarding location where the communication device that is the communication source is provided, and transmit boarding safety information for the identified boarding location.

[0138] The manner of notification to the user is not particularly limited as long as it is appropriate. For example, the wearable terminal 100 may notify the user by outputting sound / voice from a speaker. Alternatively, the wearable terminal 100 may notify the user by vibrating a vibrator. Alternatively, the wearable terminal 100 may notify the user by displaying an appropriate message on the display 132. Alternatively, the wearable terminal 100 may be configured to accept user operations by voice recognition using the microphone 133, gesture recognition using the outer camera 111, or the like.

[0139] The operation area relates to a range in which operation can be performed by the user's actions.

[0140] 100 Wearable terminal 111 Outer camera 112 Inner camera 113 Distance measurement sensor 114 Positioning sensor 115 Acceleration sensor 116 Gyro sensor 117 Geomagnetic sensor 118 Vibrator 131 Communication device 132 Display 133 Microphone 134 Speaker 135 Processor 136 Operation input interface 137 Timer 138 Memory

Claims

1. A head-mounted display worn on a user's head, comprising: a sensor; and a processor, wherein the processor determines whether a shared place exists around the user based on data obtained from the sensor, and if it determines that a shared place exists around the user, provides notification regarding the setting of the operation area of ​​the head-mounted display.

2. A head-mounted display worn on a user's head, comprising: a sensor; and a processor, wherein the processor determines whether or not the user is in a vehicle based on data obtained from the sensor, and when it determines that the user is in a vehicle, provides notification regarding the setting of the operation area of ​​the head-mounted display.

3. A head mounted display as claimed in claim 2, characterized in that the processor uses data obtained from the sensor to determine a boarding location or a location shared by passengers in the vehicle.

4. A head-mounted display as described in claim 3, wherein the processor sets the operation area according to the user's boarding location, or sets an upper limit of the range in which the operation area can be set at the user's boarding location.

5. A head mounted display as claimed in claim 2, characterized in that the processor analyzes the size of the user's body from image data and / or distance measurement data obtained from the sensor, and adjusts the upper limit of the range in which the operation area can be set according to the size of the user's body.

6. A head-mounted display as described in claim 3, characterized in that the processor, when determining that there has been a change in the environment at the boarding location using image data and / or distance measurement data obtained from the sensor, issues a notification regarding reconfiguration of the operation area.

7. A head mounted display as described in claim 2, wherein the processor determines whether it is safe for the user to ride based on data obtained from the sensor, and sets the operation area when it is determined that the user's ride is safe.

8. A head mounted display as described in claim 2, wherein the processor determines whether it is safe for the user to ride based on data obtained from the sensor, and when it determines that it is safe for the user to ride, transitions to a mode in which setting of the operation area is not required.

9. A head mounted display as described in claim 2, wherein the processor: determines whether the user is riding in a vehicle in which wearing a seat belt is exempt based on data obtained from the sensor and / or information input by the user; if it determines that the user is riding in a vehicle in which wearing a seat belt is exempt, confirms whether the user's ride is safe based on data obtained from the sensor; and if it confirms that the user's ride is safe, transitions to a mode in which setting of the operation area is not required.

10. A head mounted display as described in claim 2, wherein the processor: determines whether the user is riding in a vehicle in which wearing a seat belt is exempt based on data obtained from the sensor and / or information input by the user; if it determines that the user is riding in a vehicle in which wearing a seat belt is not exempt, checks whether a seat belt is present in the user's seat based on data obtained from the sensor; and if it confirms that a seat belt is not present, prohibits setting of the operation area or transitions to a mode in which use of the head mounted display is prohibited.

11. A head-mounted display system comprising: a head-mounted display as described in claim 2; and an in-vehicle system that manages information inside the vehicle, wherein the head-mounted display and the in-vehicle system are capable of communicating with each other, the head-mounted display communicates with the in-vehicle system to determine whether it is safe for the user to ride, and when it is determined that it is safe for the user to ride, sets an operation area of ​​the head-mounted display.

12. A head-mounted display system as described in claim 11, characterized in that the head-mounted display determines whether the user is riding in a vehicle in which wearing a seat belt is exempt based on data obtained from the in-vehicle system, and if it is determined that the user is riding in a vehicle in which wearing a seat belt is exempt, confirms whether the user's ride is safe based on data obtained from the in-vehicle system, and if it is confirmed that the user's ride is safe, transitions to a mode in which setting of the operation area is not required.

13. A head-mounted display system as described in claim 11, characterized in that the head-mounted display determines whether the user is riding in a vehicle in which wearing a seat belt is exempt based on data obtained from the in-vehicle system, and if it is determined that the user is riding in a vehicle in which wearing a seat belt is not exempt, checks whether a seat belt is present in the user's seat based on data obtained from the in-vehicle system, and if it is confirmed that a seat belt is not present, prohibits setting of the operation area or transitions to a mode in which use of the head-mounted display is prohibited.

14. A control method using a head-mounted display equipped with a sensor and a processor, characterized in that the processor determines whether or not a user is in a vehicle based on data obtained from the sensor, and when the processor determines that the user is in a vehicle, notifies the user regarding the setting of the operation area of ​​the head-mounted display.

Images