Display device

Abstract

To provide a technique that can achieve a good balance of advanced functionality and weight reduction, and improve usability for users regarding HMD technology.SOLUTION: An HMD of an embodiment is a head-mounted display device that can display images in a user's field of view. The HMD includes a head-mounted portion that is attached to the user's head and has a display surface that displays images, and a trunk-mounted portion that communicates with the head-mounted portion and is mounted on a part of the user's trunk. The HMD measures the relative positional relationship between the head-mounted portion and the trunk-mounted portion, and based on the positional relationship, grasps a state including the position and direction of the head-mounted portion and a state including the position and direction of the trunk-mounted portion.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for a head mounted display device (HMD). [Background technology]

[0002] An HMD worn on the user's head can display images of virtual objects and the like on a transparent or non-transparent display surface. Conventional HMDs display images on the display surface in a position aligned with the world coordinate system, or in a position aligned with the direction of the user's head (corresponding HMD). Another proposed method is to display an image in a position aligned with the direction of the user's trunk, such as the torso.

[0003] An example of prior art related to the above HMD is JP 2019-28638 A (Patent Document 1). Patent Document 1 describes that a head-mounted display or the like is equipped with a head sensor that detects the direction of the head, a trunk sensor that detects the direction of the trunk, and the like, and constructs a virtual desktop consisting of two layers, a trunk layer and a head layer, and constituting a continuous information display area in a virtual space around the wearer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-28638 A Summary of the Invention [Problem to be solved by the invention]

[0005] HMDs tend to become heavier as their functions become more advanced. The weight of the HMD increases the physical burden on the user, making it difficult to use for long periods of time, for example. In order to reduce the physical burden on the user, it is necessary to reduce the weight of the HMD as much as possible.

[0006] An example of the sophistication of the HMD functions is a function that controls the display of images on the display surface according to the state of the user. For example, a function that allows the use of both image display aligned with the user's head direction and image display aligned with the trunk direction. In the example of Patent Document 1, the trunk direction measured from a sensor attached to the user's trunk is used as the display reference for the image. In the example of Patent Document 1, a sensor such as a three-axis angular velocity sensor is provided on both the HMD on the head and the user's trunk so that the two directions, the head direction and the trunk direction, can be detected.

[0007] In order to improve the functionality of an HMD, it is necessary to implement various devices including sensors. However, the heavier the device or the more devices that are implemented, the heavier the HMD becomes. In the example of Patent Document 1, it is necessary to provide two sensors of the same type, and such a configuration leaves room for improvement in terms of effective use of the sensors. When implementing an HMD, it is necessary to design it in a well-balanced manner, taking into account the perspectives of advanced functionality and weight.

[0008] The object of the present invention is to provide a technology for an HMD that can achieve a good balance between advanced functionality and weight reduction, and can improve usability for users. Problems other than those mentioned above will be described in the description of the embodiment of the invention. [Means for solving the problem]

[0009] A representative embodiment of the present invention has the following configuration: A display device according to one embodiment includes a display unit that displays an image and a detection unit that communicates with the display unit, measures a relative positional relationship between the display unit and the detection unit, grasps a state including the position and direction of the display unit based on the positional relationship, and a state including the position and direction of the detection unit, and has a world coordinate system, a detection coordinate system fixed to the detection unit, a display coordinate system fixed to the display unit, and an inertial coordinate system as coordinate systems that serve as a reference when displaying the image, the inertial coordinate system being a coordinate system whose origin moves in response to changes in the position of the display unit or the detection unit and does not follow changes in the direction of the display unit and the detection unit with respect to the arrangement position of the image, and the reference coordinate system is selected and set for each piece of information related to the image. Effect of the Invention

[0010] According to a representative embodiment of the present invention, advanced functionality and weight reduction can be achieved in a well-balanced manner, improving usability for the user. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of a display system including a head mounted display device (HMD) according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a configuration of an HMD according to the first embodiment. [Diagram 3] 2 is a diagram showing the configuration of a control unit and the like in the HMD of the first embodiment. [Figure 4] FIG. 2 is a diagram showing a main processing flow in the HMD of the first embodiment. [Diagram 5] 2 is a diagram showing a coordinate system and distance measurement of positional relationships in the HMD of the first embodiment. FIG. [Figure 6] 10 is a diagram showing calculation of positional relationship etc. in the HMD of the first embodiment. FIG. [Figure 7] FIG. 2 is a diagram showing an example of implementation of elements in the neck-mounted part of the HMD of the first embodiment. [Figure 8]FIG. 4 is a diagram showing a configuration example of connection lines in the HMD according to the first embodiment. [Figure 9] 4 is a diagram showing a configuration example of the width of a housing of the HMD according to the first embodiment. FIG. [Figure 10] FIG. 2 is a diagram showing a configuration example in which components are separated in the HMD according to the first embodiment. [Figure 11] 4 is a diagram showing an example of the configuration of display information and coordinate system information in the HMD of the first embodiment. FIG. [Figure 12] FIG. 11 is a diagram showing a configuration of an HMD according to a second embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing a configuration of an HMD according to a third embodiment of the present invention. [Figure 14] FIG. 11 is a diagram showing a configuration of an HMD according to a fourth embodiment of the present invention. [Figure 15] 13A to 13C are diagrams showing a first state and the like of a first display control example in an HMD according to a fifth embodiment of the present invention. [Figure 16] FIG. 23 is a diagram showing a second state of the first display control example in the HMD of the fifth embodiment. [Figure 17] FIG. 23 is a diagram showing a third state of the first display control example in the HMD of the fifth embodiment. [Figure 18] FIG. 23 is a diagram showing a fourth state of the first display control example in the HMD of the fifth embodiment. [Figure 19] FIG. 23 is a diagram showing an example of displaying application icons based on a head coordinate system as a second display control example in the HMD of the fifth embodiment. [Figure 20] FIG. 13 is a diagram showing an example of displaying application icons based on a neck coordinate system as a second display control example in the HMD of the fifth embodiment. [Figure 21] FIG. 13 is a diagram showing a third display control example in the HMD of the fifth embodiment. [Figure 22] FIG. 13 is a diagram showing an example of setting a display coordinate system of an app, as a fourth display control example, in the HMD of the fifth embodiment. [Diagram 23] FIG. 23 is a diagram showing an example of setting a display coordinate system for application icons, as a fourth display control example, in the HMD of the fifth embodiment. [Figure 24]13 is a diagram showing an example of setting a display coordinate system for a work tool, as a fourth display control example, in the HMD of the fifth embodiment. FIG. [Diagram 25] FIG. 13 is a diagram showing the concept of an inertial coordinate system as a fifth display control example in the HMD of the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0013] (Embodiment 1) A head mounted display device (HMD) according to a first embodiment of the present invention will be described with reference to FIGS.

[0014] The HMD of the first embodiment has a separate type configuration, and is roughly divided into two parts, a head mounting part and a neck mounting part. The components of the HMD function are implemented separately in these two parts. For example, a display device is implemented in the head mounting part, and a main sensor group, a controller, a battery, and the like are implemented in the neck mounting part. Such a separate type configuration can reduce the number of devices implemented in the head mounting part, and can omit the implementation of at least some of the sensors required for advanced functions in the head mounting part. As a result, this HMD can reduce the weight of the head mounting part mounted on the user's head compared to conventional HMDs. Therefore, this HMD provides a better fit for the user and is easier to use for long periods of time.

[0015] In addition, the position and direction (corresponding posture) of the neck mounting part and the position and direction (corresponding posture) of the head mounting part of the HMD can change independently according to the user's movement. Therefore, the HMD is provided with a means for measuring the relative positional relationship between the neck mounting part and the head mounting part. As an example of such a means, the HMD is provided with a distance measuring sensor in the neck mounting part, and measures the position and direction of the head mounting part from the neck mounting part. Then, the HMD uses the measured positional relationship to grasp both the state of the position and direction, etc. of the neck mounting part and the state of the position and direction, etc. of the head mounting part based on calculations such as rotation of the coordinate system.

[0016] This HMD can correct and convert the state (corresponding sensor data, etc.) detected by the sensor of either the head mounted part or the neck mounted part into the state (corresponding sensor data, etc.) in the coordinate system of the other of the head mounted part or the neck mounted part, using the positional relationship. That is, this HMD can convert the state in the head coordinate system detected by the sensor of the head mounted part into the state in the neck coordinate system of the neck mounted part, using the positional relationship. Also, this HMD can convert the state in the neck coordinate system detected by the sensor of the neck mounted part into the state in the head coordinate system of the head mounted part, using the positional relationship.

[0017] With such a configuration, the HMD of the first embodiment can detect both the user's head direction (the direction of the corresponding head mounted part) and the trunk direction (the direction of the corresponding neck mounted part), and can control the display of virtual objects using each of these two directions. For example, this HMD can simultaneously display an image at a position that matches the head direction in the head coordinate system, and at a position that matches the trunk direction in the neck coordinate system. Such advanced functions can improve convenience, such as by supporting the user's work.

[0018] The HMD of the first embodiment is particularly equipped with a position and orientation sensor in the neck mounted part rather than the head mounted part. This position and orientation sensor is a sensor that detects the position and direction (corresponding posture) based on the trunk including the neck and shoulders, in other words, a sensor that detects the state in the neck coordinate system. The state detected by this sensor is different from the state based on the head (corresponding head mounted part and head coordinate system) as it is. Therefore, this HMD obtains the state based on the head coordinate system from the state detected by the neck mounted part by calculating the coordinate system based on the above positional relationship.

[0019] [Display System] FIG. 1 shows a configuration of a display system including the HMD 1 of the first embodiment. In the HMD 1, the head mounting unit 1A and the neck mounting unit 1B are connected by a connection line 4. The head mounting unit 1A and the neck mounting unit 1B are connected by the connection line 4, which is a cable, so that communication and power can be supplied. Although not shown in FIG. 1, the body of a user who wears the HMD 1 exists. The head mounting unit 1A is worn on the user's head, and the neck mounting unit 1B is worn near the user's neck and shoulders. The HMD 1 may be optionally equipped with an operation device 2, which is a remote control. The user can operate the HMD 1 using the operation device 2 held in the user's hand. For example, the neck mounting unit 1B and the operation device 2 perform short-distance wireless communication. The HMD 1 may communicate with an external device. For example, the HMD 1 may be connected to a server 3A of a business operator or a PC 3B at home via a communication network. Note that X, Y, and Z directions may be used as directions for explanation, in addition to the coordinate system. The X direction corresponds to the front-back direction with respect to the user's body and the HMD 1. The Y direction corresponds to the left-right direction with respect to the user's body and the HMD 1. The Z direction corresponds to the up-down direction with respect to the user's body and the HMD 1.

[0020] The head mounted unit 1A has, for example, a glasses-like housing 10, and components such as a display device including a transmissive display surface 5 and a camera 6 are mounted on the housing 10. A part of the housing 10, for example near the left and right sides, includes a marker unit 13 for distance measurement, and the marker unit 13 has a plurality of markers 17 formed thereon to serve as measurement points.

[0021] The neck attachment unit 1B has a housing 11 and a housing 12, which are, for example, arc-shaped housings. The housing 11 is a part that is placed on the left and right of the user's neck and shoulders, and has a right housing 11R and a left housing 11L. The housing 12 connects the left and right housings 11, and is a part that is placed behind the user's neck and shoulders. The housing 11 is equipped with a distance sensor 7, a microphone 8, a speaker 9, an operation input unit 14, etc.

[0022] Unlike conventional HMDs, which measure the position, orientation, and other conditions of the HMD based on the difference in displacement from an initialized state, the HMD 1 is configured to measure the positional relationship, such as the position and orientation, of the head mounted unit 1A directly from the distance measuring sensor 7 of the neck mounted unit 1B. As a result, this HMD 1 omits the mounting of components such as a position and orientation sensor in the head mounted unit 1A, thereby reducing the weight of the head mounted unit 1A.

[0023] Sensors for measuring the position and orientation in conventional HMDs include an acceleration sensor, an angular velocity sensor (gyro sensor), a geomagnetic sensor, a GPS receiver, a camera, and a distance sensor. Note that these devices, including the camera, may be collectively referred to as sensors. The acceleration sensor measures the amount of movement of the HMD and can measure the gravitational acceleration vector when the HMD is stationary, so that the inclination of the HMD can be measured. The angular velocity sensor measures the change in the orientation of the HMD. The geomagnetic sensor estimates the orientation of the HMD based on the detection of the direction of the geomagnetic field. The GPS receiver can know the position of the HMD as latitude and longitude values ​​based on information that can be received by GPS radio waves. The camera captures feature points in the external world. The position and orientation of the HMD can be estimated from the direction of the feature points and map information. The distance sensor measures the distance to the feature points in the external world. The position and orientation of the HMD can be estimated from the distance and map information.

[0024] In the HMD 1 of the first embodiment, at least some of the sensors for measuring the state of the HMD as described above are mounted on the neck mounting part 1B instead of the head mounting part 1A. Therefore, the HMD 1 has a function for measuring the relative positional relationship between the neck mounting part 1B and the head mounting part 1A. Using this positional relationship, the HMD 1 performs conversion between data measured by the sensor of the neck mounting part 1B and data measured by the sensor of the head mounting part 1A.

[0025] The HMD 1 performs distance measurement to grasp the positional relationship with respect to the housing 10 of the head mounted unit 1A using a distance measurement sensor 7 mounted on the neck mounted unit 1B. In particular, the distance measurement sensor 7 measures the positions of the multiple markers 17 on the housing 10 as multiple feature points. The multiple feature points may be three or more points that are not on the same line. Based on the measured feature points, the neck mounted unit 1B can obtain the positional relationship between the head coordinate system of the head mounted unit 1A and the neck coordinate system of the neck mounted unit 1B (FIG. 5 described later).

[0026] Using the obtained positional relationship, the HMD1 can correct and convert the state detected by the sensor of the neck mounting unit 1B into a state based on the head coordinate system of the head mounting unit 1A. Also, using the obtained positional relationship, the HMD1 can correct and convert the state detected by the sensor of the head mounting unit 1A into a state based on the neck coordinate system of the neck mounting unit 1B. Therefore, the HMD1 can grasp both the state of the head position and direction based on the head coordinate system and the state of the trunk position and direction based on the neck coordinate system. Using the data of these states, the HMD1 can perform both display control of virtual objects according to the head direction, etc. and display control of virtual objects according to the trunk direction, etc.

[0027] [HMD] 2 shows a functional block configuration of the HMD 1 according to the embodiment 1. The head mounted unit 1A includes a processor 100, a memory 120, a communication interface unit 130, a power supply circuit 141, a sub-battery 142, a display device 150, a camera 6, an operation input unit 18, etc., which are connected to each other via a bus or the like.

[0028] The processor 100 is a sub-controller that controls the head mounted unit 1A. The memory 120 stores data and information handled by the processor 100 and the like. The communication interface unit 130 is a part such as a communication circuit that performs wired communication with the neck mounted unit 1B via a connection line 4. The power supply circuit 141 is a part that receives power from the neck mounted unit 1B via the connection line 4, and charges the sub-battery 142. The sub-battery 142 supplies power to each part in the head mounted unit 1A. The implementation of the sub-battery 142 may be omitted. In that case, power is supplied from the power supply circuit 141 to each part. The display device 150 displays a video or image of a virtual object or the like in the area of ​​the display surface 5 based on the display data. The camera 6 captures an image of the surroundings including the front of the head mounted unit 1A. The operation input unit 18 includes, for example, an operation button for operating the HMD 1. The implementation of the operation input unit 18 can be omitted. The marker unit 13 includes a plurality of markers 17, and may include a circuit for controlling the emission of the markers 17.

[0029] The neck-mounted unit 1B includes a processor 200, a memory 220, a communication interface unit 230, a power supply circuit 241, a main battery 242, an operation input unit 14, a distance sensor 7, a position and orientation sensor 70, an audio input unit, and an audio output unit, etc., which are interconnected via a bus or the like.

[0030] The processor 200 is a main controller that controls the entire HMD 1 including the neck mounting unit 1B. The memory 220 stores data and information handled by the processor 200 and the like. The communication interface unit 230 is a part such as a communication circuit that performs wired communication with the head mounting unit 1A via a connection line 4. The communication interface unit 230 also performs wireless communication with the outside and short-range wireless communication with the controller 2. The power supply circuit 241 is a part that charges the main battery 242 from an external power source and supplies power to the neck mounting unit 1B via the connection line 4. The main battery 242 supplies power to each part in the neck mounting unit 1B. The operation input unit 14 includes, for example, an operation button for operating the HMD 1. The distance measurement sensor 7 is a combined type described later, and performs both distance measurement of the marker 17 and normal distance measurement. The position and orientation sensor 70 includes an acceleration sensor 71, an angular velocity sensor 72, a geomagnetic sensor 73, and a GPS receiver 74. The audio input unit includes left and right microphones 8. The audio output unit includes left and right speakers 9 and earphones.

[0031] The memory 220 also stores a control program for configuring the functions of the HMD 1, an application program for implementing each application, and setting information. The setting information includes system setting information and user setting information. The control program or application program is, for example, a program for displaying virtual objects (corresponding display objects) including a graphical user interface (GUI) for work support in the user's field of vision.

[0032] In the first embodiment, the neck attachment unit 1B is a part attached to the user's trunk near the neck and shoulders, but it is not limited thereto. The neck attachment unit 1B may be configured to detect the state of the user's trunk, such as the orientation, and may be a trunk attachment unit attached to any part of the trunk. In other words, the trunk attachment unit may be a part attached to the chest, back, abdomen, waist, etc.

[0033] The reason why the neck area is selected as the attachment location in the first embodiment is that it is convenient for using a speaker and a microphone, and also for the following reasons. That is, the distance between the head attachment unit and the torso attachment unit is shortened as much as possible, and the length of the cable (corresponding connection line 4) when connecting them by wire is shortened as much as possible. The shorter the cable, the less the feeling of the cable clinging to the body can be reduced, and the user's wearing comfort, ease of movement, and ease of use can be improved. As a modification, the connection for communication and power supply between the head attachment unit 1A and the neck attachment unit 1B is not limited to being wired, and may be wireless.

[0034] [HMD-processing section] FIG. 3 shows a functional block configuration of a control unit and the like realized by processor 100 of head mounted unit 1A and processor 200 of neck mounted unit 1B.

[0035] The head mounted unit 1A has a data acquisition unit 100a, a data processing unit 100b, a communication control unit 100c, and a display control unit 100d as processing units within a control unit realized by a processor 100 etc. The head mounted unit 1A stores display information D11 etc. in a storage unit realized by a memory 120 etc. The display information D11 is display data etc. received from the neck mounted unit 1B.

[0036] The neck mounted unit 1B has a data acquisition unit 200a, a data processing unit 200b, a communication control unit 200c, and a display control unit 200d as processing units in the control unit realized by the processor 200, etc. The neck mounted unit 1B stores display information D21, coordinate system information D22, etc. in a storage unit realized by the memory 220, etc. The display information D21 corresponds to display data including images and control information for display control, and includes data to be transmitted to the head mounted unit 1A. The coordinate system information D22 corresponds to information for managing and controlling each coordinate system of the head coordinate system and the neck coordinate system described later, and the positional relationship between them.

[0037] The data acquisition unit 100a of the head mounted unit 1A acquires data such as images captured by the camera 6 and stores it in the memory unit. If the head mounted unit 1A is equipped with another sensor, the data acquisition unit 100a acquires data detected by that sensor and stores it in the memory unit. The data processing unit 100b processes the data acquired by the data acquisition unit 100a as necessary and passes it to the communication control unit 100c. The communication control unit 100c controls the transmission of the data to the neck mounted unit 1B via the communication interface unit 130.

[0038] The data acquisition unit 200a of the neck mounting unit 1B acquires data detected by each sensor of the position and orientation sensor 70 and data detected by normal distance measurement by the distance measurement sensor 7, and stores it in the memory unit. The communication control unit 200c receives data from the communication control unit 100c of the head mounting unit 1A through the communication interface unit 230 and passes it to the data processing unit 200b. The processor 200 also causes the distance measurement sensor 7 to measure the distance to the marker 17 of the head mounting unit 1A. The data acquisition unit 200a also acquires the distance measurement data. The data processing unit 200b processes each piece of data acquired by the data acquisition unit 200a and each piece of data from the head mounting unit 1A as necessary.

[0039] The data processor 200b calculates the relative positional relationship of the head mounted unit 1A to the neck mounted unit 1B based on the distance measurement data of the marker 17. This positional relationship corresponds to the positional relationship of the origin and the rotational relationship between the neck coordinate system and the head coordinate system. The data processor 200b stores data on the calculated positional relationship in the coordinate system information D22.

[0040] The data processing unit 200b uses the positional relationship to calculate the state of the position and direction of the neck coordinate system and the state of the position and direction of the head coordinate system. At this time, the data processing unit 200b corrects and converts the data detected by the position and orientation sensor 70 of the neck mounting unit 1B into data representing the state in the head coordinate system based on the positional relationship. At this time, the data processing unit 200b performs the correction and conversion based on a calculation of the rotation of the coordinate system, which will be described later. The data processing unit 200b stores the calculated state data in the coordinate system information D22.

[0041] The display control unit 200d refers to data such as the state obtained by the data processing unit 200b and controls the display of virtual objects on the display surface 5. For example, the display control unit 200d creates display data for displaying an image at a position aligned with the head direction and display data for displaying an image at a position aligned with the trunk direction, and stores them in the display information D22. The display control unit 200d may also control the display using image data from the camera 6 of the head mounting unit 1A.

[0042] The communication control unit 200c controls the transmission of the display data of the display information D21 to the head mounted unit 1A via the communication interface unit 230. The communication control unit 100c of the head mounted unit 1A receives the display data and stores it in the display information D11. The display control unit 100d controls the display device 150 according to the display data to display an image on the display surface 5.

[0043] In the first embodiment, the display control unit 100d is configured to directly display an image on the display surface 5 according to the display data from the display control unit 200d, so the implementation of the display control unit 100d can be omitted or simplified. In other embodiments, the display control unit 100d may perform a specific display control in the head mounting unit 1A, independent of the display control by the display control unit 200d. For example, the display control unit 100d may perform display control using an image from the camera 6. In addition, if the head mounting unit 1A has a gaze detection function, the display control unit 100d may perform image display control according to the gaze direction detected using that function. In addition, if the head mounting unit 1A is configured not to have a camera 6 or other sensors, the implementation of the data processing unit 100b can be omitted or simplified. By configuring the main processing to be performed by the main controller of the neck mounting unit 1B, the implementation of the device in the head mounting unit 1A can be reduced.

[0044] Furthermore, in embodiment 1, calculations such as state conversion using positional relationship are performed by data processing unit 200b of neck mounting unit 1B, but in other embodiments, positional relationship data may be transmitted from neck mounting unit 1B to head mounting unit 1A, and similar calculations may be performed by data processing unit 100b of head mounting unit 1A.

[0045] [HMD-Processing Flow] Fig. 4 shows a main processing flow in the HMD 1 of the first embodiment. Fig. 4 includes steps S11 to S17 by the neck mounting unit 1B and steps S21 to S26 by the head mounting unit 1A. In step S11, when the power of the neck mounting unit 1B is turned on based on the user's operation, the neck mounting unit 1B performs a startup process to start circuit operation and communicates with the head mounting unit 1A. In step S21, the head mounting unit 1A starts circuit operation in response to communication from the neck mounting unit 1B.

[0046] In step S12, the neck mounted unit 1B acquires each piece of data by the data acquisition unit 200a. The acquired data at this time includes detection data by the position and orientation sensor 70 and distance measurement data of the marker 17 by the distance measurement sensor 7. In step S22, the head mounted unit 1A acquires data by the data acquisition unit 100a. In step S23, the head mounted unit 1A transmits the data to the neck mounted unit 1B by the communication control unit 100c. In step S13, the neck mounted unit 1B receives the data by the communication control unit 200c.

[0047] In step S14, the data processing unit 200b uses the acquired data to calculate the positional relationship with the head mounted unit 1A, and uses the positional relationship to perform correction and conversion on the state detected by the position and orientation sensor 70. As a result, the data processing unit 200b obtains both data representing the state in the neck coordinate system and data representing the state in the head coordinate system, and updates the coordinate system information D22.

[0048] In step S15, the display control unit 200d creates display data for display control of the virtual object based on the coordinate system information D22, and stores it in the display information D21. In step S16, the communication control unit 200c transmits the display data to the head mounted unit 1A. In step S24, the communication control unit 100c of the head mounted unit 1A receives the display data. In step S25, the display control unit 100d displays an image on the display surface 5 based on the display data.

[0049] In step S17, if the power supply of the neck mounting unit 1B is turned off based on the user's operation (Yes), the neck mounting unit 1B performs a shutdown process for the neck mounting unit 1B, and this flow ends. During the shutdown process, communication with the head mounting unit 1A is also performed. If the power supply remains on (No), the process returns to step S12, and the same process is repeated. Also, in step S26, if the power supply of the neck mounting unit 1B is turned off (Yes), the head mounting unit 1A performs a shutdown process for the head mounting unit 1A, and this flow ends. If the power supply remains on (No), the process returns to step S22, and the same process is repeated.

[0050] [Coordinate System and Geometry] FIG. 5 shows the configuration related to each coordinate system and positional relationship in the HMD 1 of the first embodiment. Note that FIG. 5 omits the illustration of the connection line 4 and the user's body. (A) shows a first example of the state of the HMD 1, and (B) shows a second example of the state. In the state (A), the direction of the user's trunk such as the neck and the head is almost the same and faces forward. Correspondingly, the direction (axis XN) of the neck mounting unit 1B and the direction (axis XH) of the head mounting unit 1A are almost the same. The coordinate system in the head mounting unit 1A is the head coordinate system CH {axes XH, YH, ZH}, and the origin is the origin OH. The coordinate system in the neck mounting unit 1B is the neck coordinate system CN {axes XN, YN, ZN}, and the origin is the origin ON. The position of the origin ON is determined based on the relative positional relationship from each part of the neck mounting unit 1B (particularly the position of the distance measuring sensor 7). In this state, the axes XH and XN correspond to the forward direction, the axes YH and YN correspond to the leftward direction, and the axes ZH and ZN correspond to the upward direction. The neck coordinate system CN is, in other words, the trunk coordinate system.

[0051] Vector V1 is a vector that represents the relative positional relationship of head mounted unit 1A to neck mounted unit 1B, and is a vector from origin ON to origin OH. Note that if a vector in the opposite direction to vector V1 is considered, that vector represents the relative positional relationship of neck mounted unit 1B to head mounted unit 1A.

[0052] The state of (B) is an example of a case where the user changes the orientation and position of the head from the state of (A) without changing the trunk of the body much. In the state of (B), the state of the neck mounting unit 1B has hardly changed compared to the state of (A). In the state of (B), the position of the head mounting unit 1A has moved slightly forward compared to the state of (A), and the direction of the head mounting unit 1A has rotated slightly to the left.

[0053] The HMD 1 measures the distance from the distance sensor 7 of the neck mounted part 1B to the multiple markers 17 of the head mounted part 1A. The vector v1 is a vector from the position of the distance sensor 7 to the position of the marker 17 during the distance measurement. The HMD 1 calculates the relative positional relationship of the head mounted part 1A from the neck mounted part 1B based on the distance measurement data. This positional relationship is expressed by the vector V1 between the origins and the rotation between the head coordinate system CH and the neck coordinate system CN.

[0054] The measurement points (corresponding markers 17) that need to be measured in order to calculate the positional relationship are at least three points that are not on the same line. In the illustrated example, the two distance measuring sensors 7 on the left and right are used to measure the distance to a total of four points on the left and right, but this is not limited to this, and the positional relationship can be calculated as long as the distance to three points can be measured as a result. In the illustrated example, the markers 17 are located at one point at the front end of the side surface of the housing 10 and one point along the way, but this is not limited to this and is possible. Also, in this configuration example, the two distance measuring sensors 7 on the left and right measure the distance to two points on the left and right of the housing 10, respectively. This is not limited to this, and the distance to the left and right markers 17 of the housing 10 may be measured from one distance measuring sensor 7.

[0055] In FIG. 5, the distance measurement sensor 7 is a dual-use type, and is capable of two types of distance measurement: distance measurement 501 for calculating positional relationship and normal distance measurement 502. The distance measurement 501 is a distance measurement of a marker 17 including a vector v1. The normal distance measurement 502 is a distance measurement for a surrounding object including the front of the HMD 1. The normal distance measurement 502 is, for example, a distance measurement for grasping the structure of a surrounding room. Not limited to this, in the HMD of the modified example, the distance measurement sensor for the distance measurement 501 for calculating positional relationship and the distance measurement sensor for the normal distance measurement 502 may be provided separately. For example, the distance measurement sensor for the normal distance measurement 502 may be provided at the front position of the housing 11, and the distance measurement sensor for the distance measurement 501 may be provided at the middle or rear position of the housing 11 or the housing 12. Moreover, the two types of distance measurement sensors may be separately implemented in the head mounting unit 1A and the neck mounting unit 1B.

[0056] [Distance sensor] The distance measuring sensor 7 may be a TOF sensor (TOF: Time Of Flight) or a stereo camera. The TOF sensor detects the light that is returned after the irradiated reference light hits an object, and calculates the distance from the time of flight of the light. When the TOF sensor is used, a reflective member that efficiently reflects light of the wavelength of the reference light may be arranged on the head mounted unit 1A as a marker 17. The wavelength of the reference light is, for example, a wavelength in the near infrared range. This increases the intensity of the reflected light of the TOF sensor, reducing measurement errors and making it easier to detect feature points. In addition, when a reference light in the near infrared range is used, it also has the effect of being resistant to the influence of ambient light outdoors.

[0057] A stereo camera calculates distance from the difference between feature points in the captured left and right images. When a stereo camera is used, a light-emitting marker may be disposed as the marker 17 of the head mounted unit 1A. The marker unit 13 may control the emission of the marker 17. The light-emitting marker emits, for example, near-infrared light. This increases the incident light in the stereo camera, reducing measurement errors and making it easier to detect feature points, and provides the same effects as the TOF camera.

[0058] Also, when measuring the distance from the distance measuring sensor 7 of the neck mounting unit 1B to the marker 17 of the head mounting unit 1A, some influence may make it difficult to measure the distance. In that case, the HMD 1 may measure any feature point in the outside world other than the marker 17 using both the distance measuring sensor 7 on the neck mounting unit 1B side and the camera 6 or other sensor on the head mounting unit 1A side, and estimate the positional relationship based on both of these measurement data. As a variant, the distance measuring sensor 7 may be provided on both the neck mounting unit 1B and the head mounting unit 1A, in which case the positional relationship can be estimated using distance measurement data from the two distance measuring sensors 7, upper and lower.

[0059] The number and weight of devices to be mounted can be reduced by using a dual-use distance measuring sensor 7. If the distance measuring sensor 7 is a separate sensor for normal distance measurement and for measuring the positional relationship, the number and weight of devices to be mounted increases, but there is an advantage in that the optimal distance measuring sensor can be selected according to the measurement distance.

[0060] [Positional relationship calculation and display control] 6 shows an example of the process of calculating the positional relationship described above. In step S01, the neck mounted unit 1B obtains distance measurement data DM of the distance of the marker 17 measured by the distance measurement sensor 7. In step S02, the neck mounted unit 1B uses the distance measurement data DM to calculate the positional relationship with the head mounted unit 1A and obtains positional relationship data DL. In step S03, the neck mounted unit 1B obtains data such as sensor data SN representing the state of position and direction based on the neck coordinate system CN by the position and orientation sensor 70. In step S04, the neck mounted unit 1B obtains data such as sensor data SH representing the state of position and direction based on the head coordinate system CH by performing correction and conversion processing based on the rotation of the coordinate system from the sensor data SN of the neck coordinate system CN using the positional relationship data DL.

[0061] In step S05, the neck mounted unit 1B performs display control using the sensor data SN in the neck coordinate system CN and display control using the sensor data SH in the head coordinate system CH. For example, the neck mounted unit 1B creates display data for displaying virtual objects aligned with the head direction and display data for displaying virtual objects aligned with the trunk direction, and transmits them to the head mounted unit 1A.

[0062] In step S06, head mounted unit 1A displays each image on display surface 5 based on the display data from neck mounted unit 1B. That is, head mounted unit 1A displays an image at a position aligned with the trunk direction according to the display data based on neck coordinate system CN, and also displays an image at a position aligned with the head direction according to the display data based on head coordinate system CH.

[0063] [Neck attachment part] 7 shows an example of the components mounted in the neck mounted unit 1B. In the neck mounted unit 1B, a control circuit 701 and a battery 702 are mounted in a symmetrical arrangement inside each of the left and right housings 11 (11L, 11R) in the Y direction. In the housing 11, the battery 702 is disposed on the front side in the X direction, and the control circuit 701 is disposed on the rear side. The control circuit 701 corresponds to a portion such as an IC board on which the processor 200, memory 220, communication interface unit 230, and other units in FIG. 2 are mounted. The battery 702 corresponds to a portion on which the main battery 242 is mounted.

[0064] A certain component is disposed at least in one of the left and right housings 11 or the rear housing 12. Alternatively, a certain component may be disposed as a plurality of parts divided between the left and right housings 11. A certain component may be divided between the left and right housings 11 and duplicated. For example, the processor 200 is disposed in one of the left and right control circuits 701, and another component is disposed in the other control circuit 701. For example, the main battery 242 is disposed as two parts divided between the left and right batteries 702 and duplicated. One battery 702 may be used for current use, and the other battery 702 may be used for charging. The left and right batteries 702 may be configured to be replaceable. In this case, the user can replace the other battery 702 without turning off the power while using one battery 702. Also, for example, the distance measurement sensor 7 is disposed as two distance measurement sensors divided between the left and right housings 11. The distance measurement sensor 7 and other sensors may be duplicated so that only one of the left and right sensors can function.

[0065] As in this example, the weight of each component is balanced between the left and right sides of the entire HMD 1 by making the configuration as symmetrical as possible with respect to the mounting of each component. That is, the weight of the right housing 11R and the weight of the left housing 11L are made approximately the same. The weight is also balanced in the front-rear direction including the housing 12. This improves the wearing comfort of the HMD 1 for the user. In the case of a redundant configuration in which a certain component is duplicated in the left and right housings 11, effects such as improved availability can be obtained. Note that the arrangement is not limited to that shown in FIG. 7, and for example, the control circuit 701 may be arranged on the front side in the X direction and the battery 702 on the rear side. Other components such as the microphone 8 and the speaker 9 are also arranged on the left and right housings 11, respectively, so as to balance the weight between the left and right sides.

[0066] The power consumption of the HMD tends to increase with the sophistication of its functions. Accordingly, the battery capacity and weight required when a battery is installed also tend to increase. Therefore, in the example of the first embodiment, the weight of the head mounting unit 1A is reduced by mounting the main battery 242 in the neck mounting unit 1B. At the same time, the weight of the neck mounting unit 1B is balanced between the left and right sides by symmetrically arranging parts of the same weight as the two batteries 702 in the left and right housings 11 of the neck mounting unit 1B. The left and right weight balance may be achieved by arranging the same parts on the left and right sides, or between a certain component (for example, the processor 200) and another component. This makes it easier to balance the weight of the entire HMD 1, and also reduces the cause of positional deviation of the neck mounting unit 1B when worn.

[0067] Connection Line FIG. 8 shows a configuration example regarding the attachment of the connection wire 4 in the HMD 1. (A) is a first configuration example, which is a configuration example considering ease of wearing. The HMD 1 has left and right connection wires 4 (4L, 4R). One end of the connection wire 4 is connected to a terminal located near the center of the left and right parts of the housing 10 of the head mounting unit 1A, in front of the user's ears when wearing it. The other end of the connection wire 4 is connected to a terminal located near the center of the left and right housings 11 of the neck mounting unit 1B. The connection wire 4 is in a state of hanging down in front of the user's ears. The interval 801 is the distance between the attachment positions of the left and right connection wires 4 (4L, 4R) of the housing 10 of the head mounting unit 1A in the Y direction. The interval 802 is the distance between the attachment positions of the left and right connection wires 4 of the housing 11 of the neck mounting unit 1B in the Y direction. In the HMD 1, the positional deviation of the display surface 5 with respect to the head affects the display performance. For this reason, the head mounting part 1A and the neck mounting part 1B are connected symmetrically by two connecting wires 4. This is preferable because it balances the weight between the left and right sides and reduces the cause of positional deviation.

[0068] When wearing the HMD 1, the user places the neck mounting unit 1B around the neck and shoulders, and the head mounting unit 1A in front of the eyes of the head. When the user is not using the image display by the head mounting unit 1A, such as when resting, the user can remove the head mounting unit 1A from the head and hang it down in front of the chest. Note that in this configuration example, the user can connect and disconnect the connection wire 4 to the terminal of the housing, but the connection wire 4 may be fixedly connected to the housing.

[0069] (B) shows a second configuration example. One end of each of the left and right connection lines 4 (4L, 4R) is connected to the rear end positions of the left and right parts of the housing 10 of the head mounting unit 1A, at a position behind the user's ears when worn. The other end of each connection line 4 is connected to a position toward the rear of the left and right housings 11 of the neck mounting unit 1B. The connection lines 4 hang down after passing behind the user's ears. Distance 803 and distance 804 similarly indicate the distance between the attachment positions of the connection lines 4.

[0070] The form of the connection line 4 is not limited to the example in Fig. 8. In other forms, the other end of the connection line 4 may be connected to the housing 12 at the rear of the neck attachment part 1B, or the connection line 4 may be combined into one line 4 without being divided into left and right.

[0071] Regarding the spacing between connection lines 4, (A) shows a first configuration example in which spacing 801 and spacing 802 are roughly the same. (B) shows a second configuration example in which spacing 804 is slightly wider than spacing 803.

[0072] Regarding the configuration such as the distance between the housing of the HMD1 and the connection line 4, various modes are possible. The distance 801 on the head side and the distance 802 on the neck side may be the same. The distance 802 on the neck side may be made larger than the distance 801 on the head side. The distance 802 on the neck side may be made smaller than the distance 801 on the head side. Also, although the above distance is designed to be fixed in advance, as another form, for example, by making the housing have a flexible or flexible configuration or a deformable configuration, the user may be able to adjust the above distance. For example, in the neck mounting portion 1B, the left and right housings 11 may be composed of rigid members, and the rear housing 12 may be composed of a bendable member. The user can adjust the distance 802 of the neck mounting portion 1B according to their physique, clothing, wearing feeling, etc. Also, when adjusting the distance 802, the HMD1 may measure the distance 802 at that time using, for example, a distance measuring sensor 7 or other sensors. Regarding the positional relationship between the above distances 801, 802 and the components such as various sensors arranged on the housing of the HMD1, they are set as preset values for realizing the functions of the HMD1. When the above distance 802 etc. is changed, the above set value is also updated.

[0073] The neck mounting portion 1B is not limited to an arc shape and is possible. For example, the front of the housing 11 of the neck may have a closed shape. Although the left and right housings 11 are rod-shaped, it is not limited to this, and it may have a shape close to flat like a shoulder pad. The neck mounting portion 1B may have a shape of clothing such as a vest or work clothes.

[0074] FIG. 9 shows a configuration example regarding the distance between the left and right portions of the housing of the HMD1 in the Y direction in relation to FIG. 8. FIG. 9 schematically shows the front view of the state where the user wears the head mounting portion 1A and the neck mounting portion 1B. The width W1 indicates the width in the Y direction of the left and right portions of the housing 10 of the head mounting portion 1A. The width W2 indicates the width in the Y direction of the left and right portions of the housing 11 of the neck mounting portion 1B. (A) shows the case where the width W1 and the width W2 are about the same. (B) shows the case where the width W2 is larger than the width W1 (W1 < W2).

[0075] It is desirable that the positions of the distance measurement sensor 7 and marker 17, etc., and the dimensions of the widths of each part of the HMD 1 are designed so that it is easy to measure the distance from the distance measurement sensor 7 of the neck mounting part 1B to the marker 17 of the head mounting part 1A. In the case of (A), the marker 17 is measured upward from the distance measurement sensor 7. In the case of (B), the marker 17 is measured in a slightly diagonal direction upward from the distance measurement sensor 7. In the first embodiment, taking into consideration ease of mounting, reduction of the cause of positional deviation, ease of distance measurement, etc., the width W2 on the neck side is set to be equal to or larger than the width W1 on the head side (W1≦W2). This is not restrictive, and in other embodiments, the width W2 on the neck side may be smaller (W1>W2).

[0076] The marker 17 may be disposed on the downward surface or the horizontal surface of the housing 10. The distance measuring sensor 7 may be disposed on the upward surface or the horizontal surface of the housing 11. As a modified example, in (A), the distance measuring sensor 7 or other sensor may be disposed at a position 901 that extends outwardly to the left and right from the housing 11 of the neck attachment unit 1B and is spaced apart. The length of this extension may be adjustable by the user.

[0077] [Separate configuration example] FIG. 10 shows a table summarizing several configuration examples in which components are separately mounted on the top and bottom of the head mounted part 1A and the neck mounted part 1B in the HMD 1 of the first embodiment and each of the modifications.

[0078] First, the configuration in the first embodiment is as shown in FIG. 2 and the like, but the neck mounting unit 1B is provided with the distance sensor 7, the position and orientation sensor 70, the main battery 242, the microphone 8, the speaker 9, and the like, and the head mounting unit 1A is provided with the camera 6, the display device 150, the sub-battery 142, and the like. The position and orientation sensor 70 includes the acceleration sensor 71, the angular velocity sensor 72, the geomagnetic sensor 73, and the GPS receiver 74 described above, and the distance sensor 7 and the camera 6 are considered separately. Since the neck mounting unit 1B is fixed to the user's trunk, the position and orientation sensor 70 detects the state of the user's trunk, such as the direction of the trunk. In normal distance measurement, the distance sensor 7 measures the distance to objects around the neck mounting unit 1B (FIG. 5). It is advantageous for the camera 6 to capture the outside world close to the user's viewpoint and easy to capture the outside world. Therefore, in the first embodiment, the camera 6 is provided in the head mounting unit 1A, which is located higher than the neck mounting unit 1B.

[0079] Next, in the first modification of the first embodiment, a configuration point different from the first embodiment is that the camera 6 is also provided on the neck mounting unit 1B. This allows the weight of the head mounting unit 1A to be further reduced. In the first modification, the image of the camera 6 is an image based on the neck coordinate system CN. Therefore, when it is desired to obtain image information based on the head coordinate system CH, the image of the camera 6 may be corrected and converted using the positional relationship described above. In the first modification, emphasis is placed on reducing the weight of the head mounting unit 1A, and many of the components are mounted on the neck mounting unit 1B.

[0080] Next, in the configuration of Modification 2, which is different from Embodiment 1, a distance measuring sensor for normal distance measurement is provided as a second distance measuring sensor in head mounting unit 1A in addition to distance measuring sensor 7 in neck mounting unit 1B. Since it is advantageous to measure the distance to an object in the outside world when the outside world is easily visible, in Modification 2, the second distance measuring sensor is provided in a higher position, head mounting unit 1A.

[0081] Next, in the third modification, a configuration point different from the first embodiment is that some of the sensors of the position and orientation sensor 70 are provided in the head mounting unit 1A, and the other sensors are provided in the neck mounting unit 1B. In particular, an acceleration sensor 71 and an angular velocity sensor 72 are mounted on the head mounting unit 1A, and a geomagnetic sensor 73 and a GPS receiver 74 are mounted on the neck mounting unit 1B. Since the accuracy of detecting the rotation direction of the head and the direction of gravity is particularly important, when emphasis is placed on advanced functionality, it is advantageous to mount the sensors on the head mounting unit 1A as in the third modification.

[0082] Next, in the fourth modification, position and orientation sensors are provided in both the neck mounting unit 1B and the head mounting unit 1A, although the types are overlapping. In particular, a high-performance sensor is mounted as a main sensor in the neck mounting unit 1B, and a lightweight sensor is mounted as a sub-sensor in the head mounting unit 1A. For example, the HMD 1 may normally obtain high-precision data using the sensor in the neck mounting unit 1B, and obtain sensor data in the head mounting unit 1A when the sensor data in the neck mounting unit 1B cannot be obtained for some reason. The fourth modification is a configuration that emphasizes high precision and availability while suppressing the weight of the head mounting unit 1A.

[0083] As another modification, the microphone 8 and the speaker 9 may be provided on the head mounted part 1A instead of the neck mounted part 1B. Of course, it is also possible to combine the various configurations described above.

[0084] Coordinate System The relationship between the head coordinate system CH of the head mounted unit 1A and the neck coordinate system CN of the neck mounted unit 1B in Fig. 5 will be described below. In the following description, the coordinate system is unified to the right-handed system. The rotation of the coordinate system between the head coordinate system CH and the neck coordinate system CN can be expressed by Euler angles or normalized quaternions. In the following, normalized quaternions are used to express the rotation of the coordinate system. The processor 200 (particularly the data processing unit 200b in Fig. 3) performs calculations including such rotation of the coordinate system.

[0085] A normalized quaternion is a quaternion with a norm of 1, and can represent rotation around a certain axis. The normalized quaternion q that represents a rotation of angle η around the unit vector (nX, nY, nZ) as the rotation axis is given by the following formula 1. Equation 1: q=cos(η / 2)+nXsin(η / 2)i+nYsin(η / 2)j+nZsin(η / 2)k

[0086] i,j,k are the units of the quaternion. Clockwise rotation in the direction of the vector (nX,nY,nZ) is the positive rotation direction for η. Rotation of any coordinate system can be expressed by this normalized quaternion.

[0087] To summarize the usage of symbols, Sc(q) represents the real part of quaternion q. q* represents the conjugate quaternion of quaternion q. [·] defines the operator that normalizes the norm of quaternion q to 1. If q is an arbitrary quaternion, the following equation 2 is the definition of the operator [·]. The denominator on the right-hand side of equation 2 is the norm of quaternion q. Equation 2: [q]=q / (qq*)1 / 2

[0088] Next, the quaternion p representing the coordinate point or vector (pX, pY, pZ) is defined by the following equation 3. Equation 3: p=pXi+pYj+pZk

[0089] In this specification, unless otherwise specified, symbols representing coordinate points and vectors that are not component representations are assumed to be quaternion representations, and symbols representing rotations are assumed to be normalized quaternions.

[0090] Let PT(n) be the projection operator of a vector onto a plane perpendicular to the direction of unit vector n. The projection of vector p is expressed by the following equation 4. Equation 4: PT(n)p=p+nSc(np)

[0091] Suppose that a coordinate point or direction vector p1 is transformed into a coordinate point or direction vector p2 by a rotation operation around the origin represented by the quaternion q. Then, the direction vector p2 can be calculated using the following formula 5. Equation 5: p2=qp1q*

[0092] Let R(n1,n2) be the normalized quaternion that rotates unit vector n1 around an axis perpendicular to the plane containing n1 and n2 so that it overlaps with unit vector n2. R(n1,n2) is given by the following formula 6. Equation 6: R(n1,n2)=[1-n2n1]

[0093] Next, the relationship between the two coordinate systems, the head coordinate system CH of the head mounted unit 1A and the neck coordinate system CN of the neck mounted unit 1B, will be described. When the user moves his / her neck or head, the positional relationship between the two coordinate systems changes. For example, the state changes from (A) in FIG. 5 to (B). The HMD 1 of the first embodiment grasps the positional relationship between the two coordinate systems by constantly measuring the feature points indicated by the markers 17 of the head mounted unit 1A using the distance measuring sensor 7 of the neck mounted unit 1B.

[0094] The coordinates of the measurement points (corresponding markers 17) in the head coordinate system CH, in other words, the shape of the housing 10, etc., are known in advance from the design. The coordinate values ​​of these three points in the head coordinate system CH are pH0, pH1, and pH2. The coordinate values ​​of the three points in the neck coordinate system CN are known by measurement using the distance measuring sensor 7. The coordinate values ​​of these three points are pN0, pN1, and pN2.

[0095] First, the HMD1 calculates the rotation to align the direction of the coordinate system. The three measurement points are the first measurement point, the second measurement point, and the third measurement point. The unit vectors in the directions from the first measurement point to the second measurement point and the third measurement point are expressed in the head coordinate system CH as nH1 and nH2. Similarly, the unit vectors in the neck coordinate system CN are expressed as nN1 and nN2. Specifically, these are expressed by the following formula 7. Note that the number of measurement points is not limited to three, as long as two directional vectors with different orientations can be obtained. Formula 7: nH1 = [pH1 - pH0] nH2 = [pH2 - pH0] nN1=[pN1-pN0] nN2 = [pN2 - pN0]

[0096] First, in the rotation in the head coordinate system CH, consider a rotation qTA that overlaps nH1 with nN1. Here, the rotation qTA is expressed by the following formula 8. Equation 8: qTA=R(nH1,nN1)

[0097] Next, the directions in which nH1 and nH2 are rotated by this rotation qTA are defined as nA1 and nA2. These directions are expressed by the following formula 9. Formula 9: nA1=qTAnH1qTA*=nN1 nA2=qTAnH2qTA*

[0098] Since these angles are between the same directions, the angle between nA1 and nA2 is equal to the angle between nN1 and nN2. Also, since the three measurement points are not on the same line, the angle between nN1 and nN2 is not 0. Therefore, nA1, i.e., the rotation qTB around the axis of nN1, is uniquely determined, and nA2 can be superimposed on nN2. Specifically, the rotation qTB is given by the following formula 10. Equation 10: qTB=R([PT(nN1)nA2],[PT(nN1)nN2])

[0099] By this rotation qTB, nA1 and nA2 are rotated to nN1 and nN2 in the following equation 11. Equation 11: nN1=qTBnA1qTB* nN2=qTBnA2qTB*

[0100] Again, rotation qT is defined by the following formula 12. This rotation qT is the rotation in the head coordinate system CH that aligns the direction of the neck coordinate system CN with the direction of the head coordinate system CH. Equation 12: qT=qTBqTA

[0101] Finally, the relationship of the coordinate origin is calculated. If the coordinate origin of the neck coordinate system CN in the head coordinate system CH is ONH, ONH is given by the following formula 13. Equation 13: ONH=pH0-qT*pN0qT

[0102] From the above, if the positions of three or more feature points (corresponding markers 17) that are not on the same line among the feature points of the head mounted part 1A can be measured from the neck mounted part 1B, the relationship between the direction of the neck coordinate system CN and the direction of the head coordinate system CH, i.e., the above-mentioned positional relationship, can be obtained. This allows the HMD 1 to use the head coordinate system CH and the neck coordinate system CN in an integrated manner.

[0103] If the relationship between the head coordinate system CH and the neck coordinate system CN calculated as above is known, the values ​​of one coordinate system can be converted to the values ​​of the other coordinate system, regardless of whether the position and orientation sensor is arranged on the head mounting unit 1A or the neck mounting unit 1B. Therefore, the above-mentioned position and orientation sensor 70, specifically the acceleration sensor 71, the angular velocity sensor 72, the geomagnetic sensor 73, and the GPS receiver 74, the camera 6, and the distance measurement sensor 7, may basically be mounted on either the head mounting unit 1A or the neck mounting unit 1B. The HMD 1 of the first embodiment is configured to mount at least some of the above devices on the neck mounting unit 1B, not on the head mounting unit 1A. This allows the weight of the head mounting unit 1A to be reduced along with the advanced functions using the above devices.

[0104] [Display and coordinate system information] 11 shows an example of the configuration of display information D21 and coordinate system information D22 in the neck mounted unit 1B. The DB (database) of the display information D21 includes a display object information table T01. The display object information table T01 stores information such as an object ID, a display coordinate system, a placement coordinate, and a placement direction for each display object to be displayed on the display surface 5. The display coordinate system can be selected from each coordinate system. The placement coordinate and the placement direction are position coordinates and direction in the display coordinate system.

[0105] The DB of the coordinate system information D22 includes a coordinate system information table T02. The coordinate system information table T02 stores information such as the origin position and the front direction for each of various coordinate systems. The various coordinate systems include a world coordinate system (CW), a head coordinate system CH, a neck coordinate system CN, and an inertial coordinate system (CI) described below. The origins of each coordinate system are the origin G1 of the world coordinate system CW, the origin G2 (=OH) of the head coordinate system CH, the origin G3 (=ON) of the neck coordinate system CN, and the origin G4 of the inertial coordinate system CI, and each has position coordinates based on a certain coordinate system. The front direction of each coordinate system is expressed as the direction of, for example, one axis.

[0106] [Effects, etc.] As described above, according to the HMD 1 of the first embodiment, the separation type configuration makes it possible to achieve a good balance between advanced functionality and weight reduction, thereby improving usability for the user.

[0107] As a modified example, when a camera 6 is provided on the neck mounted unit 1B side (modification example 1 in FIG. 10), a wide-angle camera may be used as the camera 6. The wide-angle camera is, for example, a camera capable of capturing an image with an angle of view of 180 degrees or more. The HMD 1 may determine and cut out a portion of the wide-angle image captured by the wide-angle camera that corresponds to the head direction of the head mounted unit 1A, and use the cut-out image for image display control based on the head direction. This configuration is particularly effective in the case of a non-transparent display surface 5. The wide-angle camera may also be used both for capturing normal images of the outside world and for measuring positional relationships.

[0108] (Embodiment 2) An HMD according to the second embodiment of the present invention will be described with reference to FIG. 12. The following describes the components of the second embodiment and the like that are different from those of the first embodiment. The HMD 1 according to the second embodiment has a distance sensor 7 on the head mounting unit 1A and a plurality of markers 17 on the neck mounting unit 1B. That is, the distance measurement of the positional relationship between the neck mounting unit 1B and the head mounting unit 1A in the second embodiment is the opposite to that in the first embodiment. The HMD 1 measures the distance from the distance sensor 7 on the head mounting unit 1A to the plurality of markers 17 on the neck mounting unit 1B as characteristic points, and calculates the positional relationship of the neck mounting unit 1B to the head mounting unit 1A based on the distance measurement data. The HMD 1 uses the positional relationship to perform correction and conversion between data of the state detected by the sensor on the head mounting unit 1A and data of the state detected by the sensor on the neck mounting unit 1B. This calculation can be realized in the same way as in the first embodiment.

[0109] FIG. 12 shows the configuration of the HMD 1 of the second embodiment. The head mounted unit 1A is equipped with a distance measuring sensor 7, a camera 6, a display device 150, and the like. The neck mounted unit 1B is equipped with a position and orientation sensor 70, a battery (main battery 242), a microphone 8, a speaker 9, and the like. In this example, the head mounted unit 1A uses a distance measuring sensor 7 that is a combined type for normal distance measuring and for positional distance measuring. The distance measuring sensors 7 are arranged as two distance measuring sensors symmetrically on the left and right sides near the front left and right ends of the housing 10, and measure the distances to a plurality of markers 17 located roughly below. In this example, a total of six markers 17 are arranged on each of the left and right housings 11 of the neck mounted unit 1B, three on each side. The vector V2 is a vector that represents the positional relationship from the origin OH of the head coordinate system CH to the origin ON of the neck coordinate system CN.

[0110] According to the HMD 1 of the second embodiment, the same effect as that of the first embodiment can be achieved. However, from the viewpoint of the weight of the head mounted unit 1A, it is more advantageous to provide the distance measuring sensor 7 in the neck mounted unit 1B as in the first embodiment. As a modification of the second embodiment, a second distance measuring sensor for normal distance measurement may be provided separately in the head mounted unit 1A. The second distance measuring sensor may be provided in the neck mounted unit 1B. Also, as a modification, the camera 6 may be mounted on the neck mounted unit 1B side.

[0111] (Embodiment 3) An HMD according to the third embodiment will be described with reference to Fig. 13. Due to the influence of the user's clothing, etc., there may be cases where the positional relationship between the head mounting unit 1A and the neck mounting unit 1B cannot be optically measured using the distance measuring sensor 7. In the third embodiment, a method for measuring the positional relationship even in such cases will be shown.

[0112] 13 shows the configuration of an HMD 1 according to embodiment 3. The HMD 1 differs from embodiment 1 in that the position and orientation sensor 70 includes sensors including an acceleration sensor 71 and a geomagnetic sensor 73 in the head mounting unit 1A and the neck mounting unit 1B.

[0113] First, the position of the rotation center of the head mounted unit 1A as seen from the neck mounted unit 1B is determined for each individual and is approximately constant, so this is taken as a fixed position. The position is set in advance in the HMD 1. Alternatively, the value when the average rotation center position can be optically measured by the distance measuring sensor 7 or the like is set in the HMD 1. For example, if the rotation center of the head mounted unit 1A is taken as the coordinate origin of the head coordinate system CH, the relationship between the coordinate systems is determined by determining the directional relationship between the coordinate systems. Next, the directional relationship between each coordinate system is determined. The vertical directional relationship can be determined from the difference in the measured values ​​of the direction of the gravitational acceleration in each coordinate system, since the gravitational acceleration can be measured by the three-axis acceleration sensor that is the acceleration sensor 71. Then, the difference in the direction between the head (corresponding head coordinate system CH) and the trunk (corresponding neck coordinate system CN) on the horizontal plane can be determined by using the geomagnetic sensor 73 provided in both the head mounted unit 1A and the neck mounted unit 1B. Although the geomagnetic sensor may experience deviations in the absolute value of the direction due to the influence of the outside world, the relative value can be calculated relatively stably. As described above, the directional relationship between the head coordinate system CH and the neck coordinate system CN can be calculated in the same way as in the first embodiment.

[0114] As described above, in the third embodiment, the relationship between directions between the coordinate systems is calculated using the direction of gravitational acceleration in each coordinate system and, for example, the north direction in the geomagnetic field, instead of the directions from the first measurement point to the second measurement point and the third measurement point among the three measurement points described in the first embodiment. As a result, in the third embodiment, the positional relationship between the head mounting unit 1A and the neck mounting unit 1B can be estimated even in a situation where the positional relationship cannot be optically measured.

[0115] As described above, from the viewpoint of placing importance on advanced functionality, the third embodiment can provide the same effects as the first embodiment. As a modification of the third embodiment, the distance measuring sensor 7 may be omitted.

[0116] Other modified examples are possible as follows. The HMD 1 may be in a situation where it is temporarily unable to measure the positional relationship between the neck mounted unit 1B and the head mounted unit 1A. In that case, the HMD 1 may not be able to display an image based on the neck coordinate system CN or an image based on the head coordinate system CH, since it is unable to perform conversion using the positional relationship. In this case, the HMD 1 may perform, as an exceptional response process, a display that does not use the positional relationship, for example, a display of an image based on the head coordinate system CH or a display of an image based on the neck coordinate system CN. After that, when the HMD 1 is able to measure the positional relationship again, it stops the exceptional response process and returns to the original display.

[0117] (Embodiment 4) The fourth embodiment will be described with reference to Fig. 14. The HMD of the fourth embodiment is configured to implement a wireless power supply function for supplying power between the head mounting unit 1A and the neck mounting unit 1B, and to perform wireless communication between the head mounting unit 1A and the neck mounting unit 1B.

[0118] 14 shows an example of the configuration of the HMD 1 of the fourth embodiment. The connection line 4 is made up of one wire, and is looped by being connected to the housing 10 of the head mounted unit 1A. One end of the connection line 4 is connected to the rear end of, for example, the left side of the housing 10 of the head mounted unit 1A, and is connected to a power supply circuit 141 (FIG. 2) in the housing 10. The other end of the connection line 4 is connected to the rear end of, for example, the right side of the housing 10. This power supply circuit 141 includes a wireless power receiving circuit.

[0119] The middle part of the connection line 4 is a part that is disposed in contact with the housing 12 of the neck attachment part 1B, and has the power receiving antenna unit 4C mounted thereon. The power transmitting antenna unit 12C is mounted on the housing 12. The power transmitting antenna unit 12C is connected to a power supply circuit 241 of the housing 11. This power supply circuit 241 includes a power transmitting circuit for wireless power supply. For example, a coil is formed in each of the power transmitting antenna unit 12C and the power receiving antenna unit 4C. Wireless power supply is performed between the power transmitting antenna unit 12C and the power receiving antenna unit 4C that are close to each other by the action of electromagnetic induction. The method of wireless power supply is not limited.

[0120] A wireless communication circuit 10C corresponding to a wireless communication interface is mounted, for example, on the right side of the housing 10 of the head mounting part 1A. A wireless communication circuit 11C is mounted on a part of the housing 11 of the neck mounting part 1B, for example, a part of the right housing 11R. This HMD 1 realizes its function by linking the upper and lower parts by appropriately performing wireless communication between the wireless communication circuit 10C of the head mounting part 1A and the wireless communication circuit 11C of the neck mounting part 1B. An antenna for wireless communication may be formed on the connection line 4 or the housing 12.

[0121] The connection between the connection line 4 and the housing 12 may be configured to be detachable and fixed by the user. The connection line 4 is flexible and has a certain degree of length, allowing the user to move their neck or head freely.

[0122] (Embodiment 5) The fifth embodiment will be described with reference to Fig. 15. The HMD of the fifth embodiment has a function related to the above-mentioned display control, and shows an example of image display control according to the state of the user's head and torso.

[0123] [Display Control] The HMD of the conventional technology uses two references, a world coordinate system of the outside world and a local coordinate system aligned with the head (head coordinate system CH in the first embodiment), as a display method for images of virtual objects, etc. Alternatively, as in the example of Patent Document 1, there is a proposal to use two references, a coordinate system aligned with the head and a coordinate system aligned with the trunk (neck coordinate system CN in the first embodiment). In contrast to these, the HMD1 of the fifth embodiment uses all three references, the world coordinate system CW, the head coordinate system CH, and the neck coordinate system CN, to control image display in each coordinate system. This allows the HMD1 to realize advanced functions and improve user convenience.

[0124] In addition, in the non-transparent display surface 5 method, there is also an immersive type method in which the entire field of view of the user becomes a display that displays a VR image. In that case, if the "virtual world" in the image is interpreted as the "world" in the present invention, a similar display method can be configured and the same effect can be obtained in the mode in which the user moves through the "virtual world" and the "virtual world" is interpreted as the "world" in the present invention. Therefore, unless otherwise specified, the display method in the present invention includes both the transparent type and the non-transparent type (particularly the immersive type). In the immersive type, movement in the "virtual world" may be performed by movement in the actual outside world of the HMD, or by control input from the controller (operator 2). Either case can be regarded as movement in the world according to the user's intention.

[0125] In the HMD 1 of the fifth embodiment, for display control including control of the position at which an image is displayed on the display surface 5, the following three types of control are used in combination.

[0126] (1) Image display position control based on the world coordinate system CW This display control is used, for example, when displaying a virtual object for work support at a position near an object in the external world.

[0127] (2) Image display position control based on the head coordinate system CH. This display control is used, for example, when displaying images such as system information, menus, commands, and icons in the graphical user interface (GUI) of HMD1 within the field of view corresponding to the direction of the user's head.

[0128] (3) Image display position control based on the neck coordinate system CN. On the other hand, for displaying certain types of virtual objects, displaying them in the world coordinate system CW or the head coordinate system CH may not be convenient. Examples include displaying work procedures and work tools. If this type of virtual object is placed at a fixed position based on the world coordinate system CW, for example, it will require the user to redo the placement every time the user moves. Note that the user's movement here includes not only the user's movement in the real space, but also the movement of the user's viewpoint in the virtual space using a controller (operation device 2) or the like.

[0129] Therefore, in the fifth embodiment, the display control of this type of virtual object utilizes an arrangement based on the neck coordinate system CN. That is, the HMD1 arranges this type of virtual object at a position aligned with the direction of the user's trunk. In this arrangement, the virtual object is displayed at a position that follows the user's movement, eliminating the need to redo the arrangement.

[0130] On the other hand, when the virtual object is placed at a position based on the head coordinate system CH, it is displayed following the movement of the user, but since it is placed at a fixed position in front of the user's field of view, it may be difficult to see the front of the field of view. For example, when the virtual object is placed at the center of the front of the field of view, it is difficult to see the work target or the work in front of the user. When the virtual object is placed at a position away from the center of the front of the field of view, there is a restriction that the virtual object becomes difficult to see or a large-sized virtual object or a large number of virtual objects cannot be placed.

[0131] Therefore, in the fifth embodiment, display control based on the neck coordinate system CN is used for this type of virtual object. That is, in this display control, the target virtual object is displayed so as to follow the user's intentional movement in the world coordinate system CW and to match the trunk direction of the neck mounting unit 1B, and is displayed so as not to follow the change in the head direction. In this display control, the target virtual object is displayed so that the display position and direction on the display surface 5 change according to the change in the head direction of the head mounting unit 1A relative to the trunk direction of the neck mounting unit 1B. This provides an effect of making work easier without the above-mentioned restrictions. For example, the user can place a virtual object such as a work tool in a position that avoids the front direction where the actual object to be worked is located, and by changing the direction of the head, there is no problem in viewing the virtual object, and a large-sized virtual object or a large number of virtual objects can be placed.

[0132] [Display control example (1)] 15 to 18 show examples of display control based on each coordinate system. First, FIG. 15 (A) shows an example of the entire visual field 5A of a user wearing an HMD 1. In the center of the entire visual field 5A is a field of view (FOV) 5B, and an example of a real object is a work object 5D on a workbench 5C. The entire visual field 5A is the entire range that the user can see. The entire visual field 5A shows the entire visual field when the user is standing facing the work object 5D and the torso and face are naturally facing forward. FOV 5B shows the range, the display area, in which a display object can be displayed on the display surface 5 of a see-through HMD 1. Point P1 shows the center point of FOV 5B. (A) shows a state in which the work object 5D and the like are seen through the display surface 5 from the user's viewpoint.

[0133] In the case of a non-transparent HMD, the entire field of view 5A can be used as the display area to display the display object, and in this case, the FOV 5B corresponds to the approximate range in front of the user's head and both eyes that is easily visible. In the case of a non-transparent display surface 5, the work object 5D, etc. are displayed as virtual objects corresponding to the real thing by video see-through or simulation.

[0134] (B) shows an example in which display objects such as virtual objects are superimposed on the entire field of view 5A of (A) in the FOV 5B of the display surface 5, which is set as a first state. Elements other than the real object are display objects (corresponding images) such as virtual objects, regardless of whether the method is transparent or non-transparent. In this example, the display objects include system information 61, an HMD menu 62, a work instruction 63, and a work tool 64.

[0135] The system information 61 is information provided to the user by the system of the HMD1, and includes images showing the status of, for example, radio wave strength, time, remaining battery power, etc. The HMD menu 62 is a GUI component for the user to input instructions, etc., and is, for example, a virtual object such as a HOME button. The system information 61 and the HMD menu 62 are examples of displays based on the head coordinate system CH, and are displayed at fixed positions in the FOV5B. For example, the system information 61 is displayed in the upper side area in the FOV5B, and the HOME button, which is the HMD menu 62, is displayed in the lower left of the FOV5B. The system information 61 and the like are displayed at the same positions in the entire field of view 5A and FOV5B even if the position or direction of the HMD1 changes due to the user's movement.

[0136] The work instruction 63 is a virtual object for explaining to the user the work related to the work object 5D. The work instruction 63 is an example of a display based on the world coordinate system CW, and is displayed at a fixed position near the work object 5D. The work instruction 63 is displayed at the same position in the world coordinate system CW even if the position or direction of the HMD 1 changes due to the user's movement, etc.

[0137] The work tool 64 is a virtual object such as a tool for supporting the user's work. The work tool 64A displays a video or animation that conveys the procedure of the work. The work tool 64B is provided with a command for operation. The work instruction 63 and the work tool 64 are generated by, for example, a work support application. The work tool 64 is an example of a display based on the neck coordinate system CN, and is displayed at a position aligned with the direction of the user's trunk. When the positional relationship between the neck mounting unit 1B and the head mounting unit 1A changes, the display position of the work tool 64 changes within the entire field of view 5A. Hereinafter, a description will be given of a change in the display of the work tool 64 based on the neck coordinate system CN.

[0138] Fig. 16 shows a second state, which is an example of a state after a change from the first state in Fig. 15. Fig. 16 shows a state in which the user turns his head diagonally downward while keeping the orientation of the torso the same, so that the front of his face faces the work object 5D. There is a work instruction 63 at point P1 in the FOV 5B. The work tool 64 has moved upward in terms of the field of view and is completely out of the FOV 5B.

[0139] Fig. 17 shows a third state, which is an example of a state after a change from the first state in Fig. 15. Fig. 17 shows a state in which the user turns their face to the upper left while keeping the orientation of their torso the same, thereby lowering the work tool 64 to the lower right within the entire field of view 5B, and bringing the left work tool 64A into the FOV 5B. The work tool 64A is located at point P1.

[0140] FIG. 18 shows a fourth state, which is an example of a state after a change from the first state in FIG. 15. FIG. 18 shows a state in which the user turns the trunk diagonally forward to the left without changing the direction of the face relative to the trunk. In this case, the work instruction 63, which is a display object based on the world coordinate system CW, moves to the right within the entire field of view 5A. On the other hand, the work tool 64, which is a display object based on the neck coordinate system CN, does not change its position within the entire field of view 5A because it is positioned in accordance with the direction of the trunk. Such a display based on the neck coordinate system CN is suitable for cases in which the user uses the same work tool 64 while moving, for example.

[0141] As in the above example, in the HMD1 of the fifth embodiment, based on the detected and calculated state data, the display of virtual objects based on each of the coordinate systems of the world coordinate system CW, the head coordinate system CH, and the neck coordinate system CN can be selected and used together as a display control method for virtual objects. In particular, in the display method based on the neck coordinate system CN, it is possible to display the placement position of the virtual object in a manner that follows the movement of the user and the HMD1 but does not follow the change in the direction of the head mounting unit 1A and the head. This allows for detailed display according to the type and nature of the display object, improving convenience, such as favorably supporting the user's work.

[0142] [Display control example (2)] 19 and 20 show a second display control example in the fifth embodiment. FIG. 19 shows an example of displaying a plurality of application icons 66 in the FOV5B on the display surface 5 of the HMD 1 using a method based on the head coordinate system CH. In this example, in response to the user's selection of the HOME button on the HMD menu 62, the HMD 1 opens a home screen and displays a plurality of application icons 66 (eight in this example) in parallel on the home screen. The application icons 66 are examples of virtual objects that are GUI components, and are icons for controlling the display of corresponding applications in response to the user's selection. The HMD 1 normally uses a method of displaying the application icons 66 at fixed positions in the FOV5B based on the head coordinate system CH. When the user selects the CLOSE button on the HMD menu 62, the display returns to the original display state in which the application icons 66 are not displayed. It is more convenient for virtual objects such as the application icons 66 that are frequently used by the user to be displayed at fixed positions in the FOV5B in this way.

[0143] Note that the display of display objects that interfere with the display of the application icons 66 may be temporarily stopped, or the application icons 66 may be displayed in an upper layer in a superimposed manner. Also, for example, a method may be used in which icons are arranged on each of a plurality of pages within the FOV 5B, and a page change operation is accepted using a single page change button or the like on the HMD menu 62. This makes it possible to increase the total number of icons that can be displayed on the display surface 5.

[0144] Furthermore, FIG. 20 shows an example of displaying application icons 66 based on the neck coordinate system CN. The HMD 1 arranges multiple application icons 66 in parallel within the entire field of view 5A in accordance with the direction of the trunk. Some of the application icons 66 are displayed within the FOV 5B. The application icons 66 displayed within the FOV 5B can be changed by changing the direction of the face while keeping the direction of the trunk the same. For example, if the user turns his or her face to the upper left, the application icon 66 for "A" will be within the FOV 5B. In this way, in the display method based on the neck coordinate system CN, all of the application icons 66, which are more numerous than in the past, can be viewed within the FOV 5B by changing the direction of the head relative to the trunk.

[0145] [Display control example (3)] 21 shows a third display control example in the fifth embodiment. Furthermore, an application icon arrangement method based on the world coordinate system CW may be used. For example, an application for operating a specific device is often used at the location of the device, so it is highly convenient to arrange a virtual object of the application at a position near the device in the world coordinate system CW.

[0146] FIG. 21(A) shows an example of displaying an app icon 67 for controlling an application for operating a work object 5D at a fixed position near the work object 5D on a work table 5C in the world coordinate system CW within the entire field of view 5A.

[0147] In addition, the application icon 67 as a display object in the world coordinate system CW may be arranged at a position and orientation based on the direction of the user. In addition, in order not to interfere with the user's work, the application icon 67 may be arranged at a position that avoids the line connecting the user and the work location of the work object 5D. FIG. 21B shows an example of the arrangement of the application icon 67 as a positional relationship viewed vertically from above. The direction 2101 of the dashed-dotted arrow is a direction that connects the representative point of the user and the HMD 1 and the representative point of the work object 5D, corresponding to the head direction or the trunk direction. The direction 2102 of the dashed-dotted arrow is a direction that connects the representative point of the user and the HMD 1 and the arrangement position of the application icon 67. The arrangement position of the application icon 67 is, for example, to the right of the work object 5D so as not to overlap it. The arrangement direction of the application icon 67 is aligned with the direction 2102.

[0148] [Display control example (4)] FIG. 22 shows a fourth display control example in the fifth embodiment. The display method based on each of the above coordinate systems may be selectable according to a user's instruction or a preset setting, or may be determined and selected by the HMD 1. The display method may be selectable for each target virtual object, each type, or each application. FIG. 22 shows an example of setting a coordinate system used to display an application. FIG. 22 shows an example of displaying the properties (corresponding virtual object) of a certain application, and shows an example in which the coordinate system for displaying virtual objects such as the icon or tool of the application can be selected and set from the head coordinate system CH, the neck coordinate system CN, and the world coordinate system CW.

[0149] Moreover, Fig. 23 shows an example in which an image representing a display coordinate system is superimposed on a virtual object. In Fig. 23, an image 68 representing a display coordinate system is superimposed on a part of a certain application icon 66. The image 68 in this example is an image representing the world coordinate system CW, and indicates that the display coordinate system of this application icon 66 is the world coordinate system CW. The image 68 may be displayed at all times, or may be displayed when the user provisionally selects the application icon 66 or inputs a predetermined instruction, for example.

[0150] Furthermore, the image 68 representing the display coordinate system may be used as a control button for the display coordinate system. The HMD 1 is set so that the display coordinate system of the corresponding virtual object is switched between the head coordinate system CH, the neck coordinate system CN, and the world coordinate system CW in response to the selection of the image 68 by the user. For example, the image 68 may be a button for cyclically switching between the three coordinate systems. For example, the selection state of the image 68 of the application icon 66 represents the display coordinate system at the time of launching the corresponding application. When the application icon 66 is selected and executed, the HMD 1 launches the corresponding application at the position of the display coordinate system according to the selection state of the image 68 at that time. That is, a virtual object such as a work tool is placed at the position of the display coordinate system. Also, the selection operation of the image 68 of the application icon 66 may be used for immediate switching of the display coordinate system. For example, initially, the state of the image 68 of the application icon 66 is the head coordinate system CH, and is displayed as in the example of FIG. 19 described above. Next, the state of the image 68 is changed to the neck coordinate system CN by the selection operation of the image 68 by the user. At this time, the HMD 1 changes the coordinate system and position in which the application icon 66 is located to a position within the neck coordinate system CN. The change of the coordinate system is possible based on the relationship between the coordinate systems described above.

[0151] FIG. 24A shows an example of switching the display coordinate system of a virtual object in the example of the above-mentioned work tool 64A. This example is an example in which the display coordinate system is switched in response to a user's selection of a virtual object such as a button arranged near the virtual object. The work tool 64A has a control display object 64b that enables operations such as switching the display coordinate system displayed in an adjacent position, for example, in the upper side area, to the area of ​​a display object 64a such as a video that conveys a work procedure. The display object 64b includes a button 64c that enables operations such as switching the display coordinate system, a zoom-out button 64d indicated by "-", and an end button 64e indicated by "x".

[0152] The button 64c is a display coordinate system switching button, and may be a cyclical switching button similar to the image 68 in FIG. 23. The button 64c includes images (corresponding buttons) representing the head coordinate system CH, the neck coordinate system CN, and the world coordinate system CW. The image corresponding to the currently selected display coordinate system is highlighted, for example, by being surrounded by a frame. The user can switch the display coordinate system of this work tool 64A by selecting the button of the desired display coordinate system.

[0153] As an example of the use of display coordinate system switching, a user changes the display coordinate system of multiple display objects arranged in the world coordinate system CW at a certain location to the neck coordinate system CN or the head coordinate system CH. The user can then move from that location, as if to carry those display objects. Then, at the location after the movement, the user returns the display coordinate system of those display objects to, for example, the world coordinate system CW. This allows the user to change the position of each display object in the world coordinate system CW with little effort.

[0154] For the display objects that are being transported or that do not need to be watched for a while, the display space can be effectively utilized by reducing the display size. The HMD 1 changes the display size of the virtual object in response to a user's instruction or the like. (B) of FIG. 24 shows an example in which the work tool 64A is reduced by operating the reduce button 64d in (A). One of the control buttons is a zoom-in button 64f shown in a rectangle, and the work tool 64A can be returned to its original state by operating the zoom-in button 64f. Not limited to reduced display, it is also possible to change the display to an icon, or the like. As described above, the function of switching the display coordinate system for each virtual object can improve the user's convenience.

[0155] [Display control example (5)] FIG. 25 shows a fifth display control example in the fifth embodiment. When the positional relationship between the head mounted unit 1A and the neck mounted unit 1B is unknown, a coordinate system to be used instead of the neck coordinate system CN may be provided. Here, this coordinate system is called the inertial coordinate system CI. Specifically, this inertial coordinate system CI is a coordinate system configured by making the coordinate origin of the world coordinate system CW follow the movement of the head coordinate system CH (corresponding head mounted unit 1A) or the neck coordinate system CN (corresponding neck mounted unit 1B). For example, a display object placed at a position on the inertial coordinate system CI in the north direction as seen from the user at a certain point is always displayed at a position in the north direction with respect to the user, even if the user moves or changes his / her orientation.

[0156] Such a display method based on the inertial coordinate system CI is also a display method that follows the movement of the user and the HMD 1 with respect to the placement position of a virtual object but does not follow the change in the orientation of the head mounted unit 1A. When the front direction of the user's body (the trunk direction of the neck coordinate system CN) is unknown, using the inertial coordinate system CI has the effect of allowing a virtual object that is to be placed near the user to be placed in a wide range of directions around the user. This display method is applicable to any HMD that can measure the world coordinate system CW.

[0157] For example, when the positional relationship between the head mounted unit 1A and the neck mounted unit 1B becomes temporarily unclear for some reason and the direction of the neck coordinate system CN becomes unclear, the HMD 1 switches the display coordinate system of the display object arranged in the neck coordinate system CN to the inertial coordinate system CI. When the direction of the neck coordinate system CN becomes clear again, the HMD 1 returns the display coordinate system of the display object from the inertial coordinate system CI to the neck coordinate system CN. Alternatively, the HMD 1 may select and switch the inertial coordinate system CI as the display coordinate system of the target virtual object or the like in response to a user instruction (for example, an operation similar to that in the fourth display control example).

[0158] FIG. 25 shows the concept of the inertial coordinate system CI. Initially, a user wearing the HMD1 is at position P01 in the world coordinate system CW. The direction of the user's torso and head is, for example, north. Virtual objects 251, 252, 253, etc. are visible within the FOV 25A corresponding to the north direction. The three axes of the inertial coordinate system CI are indicated by (XI, YI, ZI,). The axis ZI corresponds to the vertical direction, the axis XI corresponds to the north direction, and the axis YI corresponds to the west direction. The HMD1 changes the display coordinate system of the virtual object 251, etc., from, for example, the neck coordinate system CN to the inertial coordinate system CI at a specified opportunity.

[0159] Position P02 shows a case where the user moves from position P01 and changes the direction of the head to rotate. The head direction is rotated about 45 degrees from the north direction to the right, for example. The origin of the inertial coordinate system CI moves from position P01 to position P02. The virtual object 251, etc., moves together with the inertial coordinate system CI and is placed in a position in the north direction from the user's position P02 while maintaining the positional relationship. Within the FOV 25B in the head direction at position P02, the virtual object 251 is not visible, but the virtual object 252 is visible. The HMD 1 returns the display coordinate system of the virtual object 251, etc., from the inertial coordinate system CI to, for example, the neck coordinate system CN at a predetermined opportunity. As described above, by introducing a new coordinate system, the display control of the virtual object can be performed under various conditions, and the convenience of the user can be further improved.

[0160] Although the present invention has been specifically described based on the embodiment, the present invention is not limited to the above-described embodiment and can be modified in various ways without departing from the gist of the present invention. [Explanation of symbols]

[0161] 1...HMD, 1A...head-mounted part, 1B...neck-mounted part, 2...operator (remote control), 3A...server, 3B...PC, 4...connecting line, 5...display surface, 6...camera, 7...distance measuring sensor, 8...microphone, 9...speaker, 10, 11, 12...housing, 14...operation input part, 17...marker.

Claims

1. A display device, comprising: The display device is configured with a first mounting part to be mounted on a head and a second mounting part to be mounted on a part other than the head, the first mounting unit includes a first processor, a first sensor, a display device, and a first communication interface unit; the second mounting unit includes a second processor, a second sensor, a display control unit, and a second communication interface unit; The second processor, receiving, by the second communication interface unit, information acquired by the first sensor and transmitted from the first communication interface unit; generating display data based on the received information and the information acquired by the second sensor by the display control unit; Transmitting the display data to the first mounting unit; The first processor displays the display data on the display device; the second mounting portion further includes a main battery, The first mounting portion further includes a sub-battery having a smaller capacity than the main battery. A display device comprising:

2. The display device according to claim 1 , When the power supply of the second mounting unit is turned on, the second mounting unit communicates with the first mounting unit and starts up the first mounting unit. A display device comprising:

3. The display device according to claim 1 , When the power supply of the second mounting unit is turned off, the second mounting unit communicates with the first mounting unit and also turns off the power supply of the first mounting unit. A display device comprising:

4. The display device according to claim 1 , power is supplied wirelessly between the second mounting portion and the first mounting portion; A display device comprising:

5. The display device according to claim 1 , Power is supplied between the second mounting unit and the first mounting unit by wire. A display device comprising:

6. The display device according to claim 1 , The second mounting unit further includes an operation input unit, controlling the second mounting unit and the first mounting unit based on an operation by a user; A display device comprising:

7. The display device according to claim 1 , The first mounting unit further includes at least one of an audio input unit and an audio output unit. A display device comprising:

8. The display device according to claim 1 , obtaining a positional relationship between the first mounting part and the second mounting part based on first measurement information obtained by measuring a position of a marker provided on the first mounting part by the second sensor of the second mounting part, and second measurement information obtained by measuring a position of a marker provided on the second mounting part by the first sensor of the first mounting part; A display device comprising:

9. The display device according to claim 8, When the marker cannot be appropriately detected by at least one of the first sensor and the second sensor, acquiring an arbitrary feature point of the external environment by at least one of the first sensor and the second sensor, and obtaining a positional relationship between the first mounting part and the second mounting part based on a position of the acquired feature point; A display device comprising:

10. The display device according to claim 1 , the display data generated by the second processor of the second mounting unit is first display data using a coordinate system based on the first mounting unit and second display data using a coordinate system based on the second mounting unit, the first processor receives the first display data and the second display data; displaying the received first display data and the received second display data on the display device; A display device comprising:

11. The display device according to claim 10, The display device includes: When the positional relationship between the first mounting portion and the second mounting portion becomes unclear, the arrangement coordinates of the second display data are From a coordinate system based on the second mounting portion, a coordinate system in which the origin of coordinates moves following a change in the position of the first mounting unit or the second mounting unit and does not follow a change in the orientation of the first mounting unit or the second mounting unit, and the coordinate system is switched to and displayed. A display device comprising:

12. The display device according to claim 11, When the positional relationship between the first mounting portion and the second mounting portion is restored, the arrangement coordinates of the second display data are From a coordinate system in which the coordinate origin moves following a change in the position of the first mounting unit or the second mounting unit and does not follow a change in the orientation of the first mounting unit or the second mounting unit, A coordinate system based on the second mounting portion, and then displaying the coordinate system back to the coordinate system. The display device according to claim 1,

Images