Mobile information terminal and object display method

Abstract

To further improve the user friendliness when a display object is displayed while utilizing a local coordinate system and the other coordinate system different from the local coordinate system in conjunction with each other.SOLUTION: A portable information terminal 1 includes a display 103, and a processor 101 that performs a display control on the display. The processor calculates coordinates where a display object is displayed by utilizing, as coordinate systems to display the display object, a local coordinate system fixed to the portable information terminal and at least one or more non-local coordinate system that is not fixed to the portable information terminal, and when displaying an enlarged region display object which is an object relating to the display object being displayed in the local coordinate system and which needs an enlarged region display out of the display region of the display, places the enlarged region display object in the non-local coordinate system.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention provides a portable information terminal and an object display method, which are particularly suitable for displaying virtual objects and are user-friendly. [Background technology]

[0002] Conventionally, there are technologies for displaying virtual objects on head-mounted displays (HMDs). Two known coordinate systems used in such virtual object display technologies are the world coordinate system and the local coordinate system.

[0003] The world coordinate system is the coordinate system of the real world. Virtual objects placed in the world coordinate system become invisible when the user moves away from that location. On the other hand, because it has the same area as the real world, a large number of virtual objects can be placed.

[0004] On the other hand, the local coordinate system is a coordinate system fixed to the HMD, and its positional relationship with the display mounted on the HMD is also fixed. From the user's perspective, virtual objects placed in the direction where the display surface exists are displayed on the display. If virtual objects are placed in the local coordinate system within the directional range where the display surface exists, the virtual objects can always be displayed and manipulated even if the user moves while wearing the HMD, because the display is also fixed to the local coordinate system. Conversely, since only virtual objects placed within the above directional range can be displayed, there is a limit to the number of virtual objects that can be placed.

[0005] Thus, conventional techniques, which only have two coordinate systems for placing virtual objects—the world coordinate system and the local coordinate system—have the problem of not being able to place a large number of virtual objects that need to be frequently referenced. Furthermore, forcibly placing virtual objects within the direction of the display surface reduces the visibility of the outside world.

[0006] As a technology to solve this problem, Patent Document 1 proposes that "a virtual object display device comprises a display and a display control device that controls the display of the display, and the display control device includes a coordinate system calculation unit that detects the movement and rotation of the virtual object display device in the real world, and defines the placement position of an inertial coordinate system virtual object using an inertial coordinate system in which the coordinate origin follows the movement of the virtual object device and the effective field of view of the display rotates within the coordinate system due to the rotation of the virtual object display device, and a display control unit that displays the inertial coordinate system virtual object within the effective field of view of the display when the inertial coordinate system virtual object is included in the effective field of view of the display (abstract excerpt)." [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] International Publication No. 2020 / 157955 [Overview of the project] [Problems that the invention aims to solve]

[0008] Patent Document 1 describes how, in addition to the local coordinate system and the world coordinate system, an inertial coordinate system can be provided as a coordinate system for placing virtual objects, thereby increasing the options for displaying objects and improving user convenience.

[0009] However, regarding virtual objects placed in a local coordinate system, there was a problem in that there was no consideration for the placement coordinate system when displaying related virtual objects that required a large display area. More specifically, when the size of a virtual object placed in a local coordinate system was changed, the virtual object that extended beyond the display could not be seen unless further actions such as scrolling the display were performed to change the display area.

[0010] The present invention has been made in view of the above circumstances, and an object thereof is to further improve user convenience when displaying a display object by using a local coordinate system and another coordinate system different from the local coordinate system in combination.

Means for Solving the Problems

[0011] In order to solve the above problems, the present invention has a configuration described in the claims. For example, a mobile information terminal, O a display for displaying an object, An operation input unit that receives operation input for the aforementioned mobile information terminal, a processor for performing display control of the display, and the processor uses a local coordinate system fixed to the mobile information terminal and a non-local coordinate system not fixed to the mobile information terminal to Note O calculate coordinates for displaying an object, The operation input unit, displayed in the local coordinate system a display object When a selection operation is performed, an object related to the display object of the non-local coordinate system to arrangement Display to do, characterized by.

Effects of the Invention

[0012] According to the present invention, it is possible to further improve user convenience when displaying a display object by using a local coordinate system and another coordinate system different from the local coordinate system in combination. Other objects, configurations, and effects than those described above will be clarified in the following embodiments.

Brief Description of the Drawings

[0013] [Figure 1] A diagram showing an example of the external configuration of an HMD. [Figure 2] A diagram showing an example of the functional block configuration of an HMD. [Figure 3] A flowchart showing the flow of processing for simple enlarged display. [Figure 4] An explanatory diagram showing the transition of the display mode in simple enlarged display. [Figure 5] A flowchart showing the flow of processing for forced enlarged area display. [Figure 6]A diagram showing a variation of a simple enlarged display. [Figure 7] A diagram showing an example of a large-scale related display. [Figure 8] A diagram showing an example of a large-scale related display. [Figure 9] A diagram showing an example of a large-scale related display. [Figure 10A] A flowchart showing the processing flow in Figure 9. [Figure 10B] A flowchart showing the processing flow in Figure 9. [Figure 11] A diagram showing an example of an evacuation sign. [Figure 12] A flowchart showing the processing flow in Figure 11. [Figure 13] A diagram showing another example of an evacuation sign. [Figure 14] A flowchart showing the processing flow in Figure 13. [Figure 15] A diagram showing an example of transparent display. [Figure 16] Diagram illustrating the thoracic coordinate system (XB, YB, ZB). [Figure 17] Diagram illustrating the inertial coordinate system (XI, YI, ZI). [Figure 18] Diagram illustrating a direction-fixed nonlocal coordinate system (XV, YV, ZV). [Figure 19] Diagram illustrating a fixed-plane nonlocal coordinate system (XP, YP). [Figure 20] Diagram illustrating a fixed-plane nonlocal coordinate system (XP, YP). [Figure 21] An explanatory diagram of the user interface according to the second embodiment. [Figure 22] An explanatory diagram of the user interface according to the third embodiment. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. Throughout the drawings, the same components and processes are denoted by the same reference numerals, and redundant explanations are omitted.

[0015] <Hardware Configuration> In the following embodiments, an HMD (Head-Mounted Display) will be used as the portable information terminal.

[0016] Figure 1 shows an example of the external configuration of HMD1.

[0017] In Figure 1, the HMD1 is equipped with a display 103 including a display area 111 in a glasses-shaped housing 10. This display 103 is, for example, a transmissive display 103, through which the real image of the outside world is transmitted to the display area 111, and an image is superimposed on that real image. The housing 10 is equipped with a controller 100, an out-camera 109, a distance measuring sensor 115, and other sensors (referred to as sensor group 110 in Figure 1), excluding the distance measuring sensor 115. In Figure 1, the distance measuring sensor 115 is shown separately from the sensor group 110, but since the distance measuring sensor 115 is also a type of sensor, in Figure 2, which will be described later, the distance measuring sensor 115 will be included in the sensor group 110.

[0018] The rear camera 109 has, for example, two cameras positioned on both the left and right sides of the housing 10, and captures images of the area including the area in front of the HMD1. The area including the area in front of the HMD1 includes the field of view of the user wearing the HMD1.

[0019] The distance measuring sensor 115 is a sensor that measures the distance between the HMD1 and an object in the outside world. The distance measuring sensor 115 may use a Time of Flight (TOF) type sensor, a stereo camera, or other methods.

[0020] The sensor group 110 includes multiple sensors for detecting the position and orientation of the HMD1. The housing 10 is equipped with an audio input device 106 including a microphone, and an audio output device 105 including a speaker and earphone jack on the left and right sides.

[0021] The HMD1 may be equipped with an operating device 20, such as a remote controller. In that case, the HMD1 communicates with the operating device 20, for example, via short-range wireless communication. The user can operate the operating device 20 by hand to input instructions regarding the functions of the HMD1, move the cursor in the display area 111, etc.

[0022] The HMD1 may communicate and interact with external devices (such as smartphones or PCs). For example, the HMD1 may receive image data of AR (Augmented Reality) objects from an application on an external device.

[0023] HMD1 may display display objects in the display area 111. For example, HMD1 generates a display object to guide the user and displays it in the display area 111. From the user's perspective, the display object displayed in the display area 111 becomes an AR object placed in an augmented reality space added to the real world that is visible through the display area 111.

[0024] Figure 2 shows an example of the functional block configuration of the HMD1 in Figure 1. In this embodiment, the HMD1 is used as an example of a mobile information terminal for explanation, but other mobile information terminals such as smartphones 5 (see Figures 19 and 20) and tablet terminals have a similar configuration.

[0025] The HMD1 includes a processor 101, memory 102, display 103, wireless communication device 104, audio output device 105 including a speaker, audio input device 106 including a microphone, operation input unit 107, battery 108, rear camera 109, and sensor group 110, etc. These elements are interconnected via a bus or the like.

[0026] The processor 101 consists of a CPU, GPU, etc., and constitutes the controller 100 of the HMD1. The processor 101 implements functions of the OS, middleware, applications, and other functions by executing processes according to the control program 31 and application program 32 stored in the memory 102.

[0027] Memory 102 consists of ROM, RAM, etc., and stores various data and information handled by the processor 101, etc. Memory 102 also stores temporary information such as images acquired by the rear camera 109 and detection information.

[0028] The rear camera 109 acquires an image by converting light entering through the lens into an electrical signal using an image sensor.

[0029] The distance measuring sensor 115, for example, when using a TOF (Time of Flight) sensor, calculates the distance to an object from the time it takes for light emitted from the outside to hit the object and return.

[0030] The sensor group 110 includes, for example, an accelerometer 141, a gyroscope (angular velocity sensor) 142, a geomagnetic sensor 143, a GPS receiver 114, and a distance measuring sensor 115. The sensor group 110 uses the detection information from these sensors to detect the position, orientation, movement, and other states of the HMD1. The HMD1 is not limited to these sensors and may also include an illuminance sensor, proximity sensor, barometric pressure sensor, etc.

[0031] The display 103 includes a display driving circuit and a display area 111, and displays a display object in the display area 111 based on the image data of the display information 34. Note that the display 103 is not limited to a transparent display, but may also be an opaque display or the like.

[0032] The wireless communication device 104 includes communication processing circuits and antennas that correspond to various predetermined communication interfaces. Examples of communication interfaces include mobile networks, Wi-Fi®, Bluetooth®, and infrared. The wireless communication device 104 performs wireless communication processing with other HMD1s and access points. The wireless communication device 104 also performs short-range communication processing with the operator 20.

[0033] The voice input device 106 converts the audio input from the microphone into audio data. The voice input device 106 may also be equipped with a voice recognition function.

[0034] The audio output device 105 outputs sound from a speaker or the like based on the audio data. The audio output device 105 may also have a speech synthesis function.

[0035] The operation input section 107 is the part that receives operation inputs to the HMD1, such as turning the power on or off or adjusting the volume, and is composed of hardware buttons, touch sensors, etc.

[0036] Battery 108 supplies power to each component.

[0037] The processor 101 includes, as an example of the configuration of functional blocks realized by processing, a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D.

[0038] Memory 102 stores control programs 31, application programs 32, setting information 33, display information 34, terminal position and orientation information 35, and the like.

[0039] Control program 31 is a program for controlling the entire HMD1, including display control.

[0040] Application programs 32 are various programs used by the user.

[0041] The configuration information 33 includes system configuration information and user configuration information related to each function.

[0042] The display information 34 includes image data and position coordinate information for displaying the display object in the display area 111.

[0043] The terminal position and orientation information 35 is information related to the movement and orientation changes of the HMD1, used to calculate the position and orientation of the mobile information terminal in a non-local coordinate system.

[0044] The communication control unit 101A controls the communication processing using the wireless communication device 104 when communicating with other HMD1s, etc.

[0045] The display control unit 101B uses the display information 34 to control the display of display objects and the like in the display area 111 of the display 103.

[0046] The data processing unit 101C reads and writes terminal position and orientation information 35 and performs calculations such as the position and orientation of the mobile information terminal in a non-local coordinate system.

[0047] The data acquisition unit 101D acquires various detection data from the rear camera 109 and sensor group 110, and generates terminal position and orientation information 35.

[0048] <First Embodiment> The first embodiment is an embodiment in which a mobile information terminal 1 receives a user's instruction to change the display of a display object placed in a local coordinate system, and when the newly displayed display object requires a larger area than the display area of ​​the local coordinate system, the newly displayed display object is placed in a non-local coordinate system and displayed.

[0049] Before describing the first embodiment, the terms used in this embodiment will be explained.

[0050] A "local coordinate system" is a coordinate system fixed to the display area 111 of the HMD1's display 103. For a user wearing the HMD1, this is the coordinate system they see in front of them, regardless of their orientation, as long as the display area 111 is positioned directly in front of them.

[0051] A "non-local coordinate system" is a coordinate system that is not fixed to the display area 111 of the display 103. For example, a world coordinate system (X) fixed to real space. W , Y W , Z W) By changing the orientation and position of HMD1, the display area of ​​the non-local coordinate system can be changed. In other words, it is a coordinate system in which what the user sees changes when they turn their head. As non-local coordinate systems, there are the world coordinate system, a coordinate system fixed in front of the user's body from the neck down, and an inertial coordinate system in which the direction the neck is on average facing is considered the front. These will be discussed later.

[0052] "Enlarged area display" refers to a display that requires an area larger than the display area 111 of the display 103. In other words, "enlarged area display" is a display that requires an "enlarged area," and the display object is not necessarily "enlarged." In the following explanation, placing a display object outside the display area 111 of the display 103 at the same size without "enlarging" it will also be referred to as "relocation."

[0053] The first embodiment will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing the processing flow of simple magnification. Figure 4 is an explanatory diagram showing the transition of display modes in simple magnification. Hereinafter, the explanation will be given using the case where the display change instruction is a "simple magnification instruction" as an example, following the steps in Figure 3.

[0054] (Simple enlarged view) Simple enlargement display is a display mode in which a display object shown in a local coordinate system is enlarged and displayed as an enlarged display object of the same content, placed in either the local or non-local coordinate system.

[0055] The HMD1's processor 101 initially displays the display object 300 in the display area 111a, as shown in the upper part of Figure 4 (S101). The coordinate system indicating the display position within the HMD1's display area 111a is a local coordinate system fixed to the HMD1. The local coordinate system is a 3-axis Cartesian coordinate system, with the left-right direction of the display area 111a being Y L The axis, the height direction of the display area 111a is Z L The axis, the depth direction perpendicular to the display area 111a is X LLet it be the axis. When restricting the display position of the display area 111a to a certain plane, if it is restricted to a certain plane, the display position in the two-dimensional coordinates (Y L , Z L ) can be represented. It is also possible to arrange the display object three-dimensionally with respect to the local coordinate system. In this case, the display position is represented by three-dimensional coordinates (X L , Y L , Z L ), and the display area visible to the user is a pyramidal area with the user's viewpoint as the apex.

[0056] The processor 101 waits for a user instruction (S102) in a state where the display object 300 is arranged in the local coordinate system (X L , Y L , Z L ) and displayed in the display area 111a. If there is no user instruction to change the display (S103: NO), it waits (S102).

[0057] When the user gives an instruction "simple enlargement display instruction" to display an enlarged display object related to the display object 300 as a display change instruction, and the processor 101 receives the input of the simple enlargement display instruction (S103: Yes), it calculates the size of the enlarged object 301 obtained by enlarging the display object 300, and determines whether it is necessary to display it outside the display area 111a (S104).

[0058] If the processor 101 determines that the enlarged object 301 cannot be fully displayed within the display area 111a and that it is necessary to display it outside the display area 111a (S104: Yes), it places the enlarged object 301 in a non-local coordinate system (e.g., the world coordinate system) as an enlarged area display object and displays it (S105). In the example in Figure 4, the enlarged object 301, which is a simple enlarged display of the display object 300, cannot be displayed unless it extends beyond the display area 111a. Therefore, the processor 101 places the enlarged object 301 in a non-local coordinate system as an enlarged area display object. Specifically, "placed in a non-local coordinate system" means that the processor 101 calculates the coordinates of the enlarged object 301 in the non-local coordinate system and stores them in the display information 34. Using the terminal position and orientation information 35 and the display information 34, if the enlarged area display object is included in the pyramidal directional position of the display area 111 of the HMD1, it is displayed on the display 103 of the HMD1.

[0059] As shown in the lower part of Figure 4, the portion of the enlarged object 301 that extends beyond the display area 111a, when placed in a non-local coordinate system, is not visible as is. Therefore, when the direction of the HMD1 is shifted towards the upper right of the page in Figure 4, the position within the non-local coordinate system changes from display area 111a to display area 111b. In display area 111b, a larger portion of the enlarged object 301 is displayed.

[0060] Processor 101 is in a non-local coordinate system (X W , Y W , Z W With the enlarged object 301 (enlarged area display object) placed in ), the system awaits user instructions (S106).

[0061] When processor 101 receives an input instructing it to end the magnified area display (S107:YES), it ends the magnified area display (S108). "Ending the magnified area display" means shrinking the magnified object 301 back to the size of the display object 300 and rearranging it in the local coordinate system. If there is no instruction from the user (S107:NO), it waits (S106).

[0062] On the other hand, if the processor 101 determines that the enlarged object can be displayed within the display area 111a (S104: NO), it displays the enlarged object while keeping it in the local coordinate system (S109).

[0063] Processor 101 is in the local coordinate system (X L , Y L , Z L With the enlarged object placed in the specified location, the system awaits user instructions (S110).

[0064] If a command to end the display of display object 300 is received (S111:YES), the display is terminated (S112). If no command to end the display is received (S111:NO), the system waits for user instructions (S110).

[0065] (Forced enlargement of area display) This section describes forced magnification area display, a variation of simple magnification display. Figure 5 is a flowchart showing the processing flow of forced magnification area display. Regardless of the size of the magnified display object, it is forcibly set to magnification area display, and the magnified display object is placed in a non-local coordinate system. This is suitable when you want to enlarge the display object and at the same time carefully view the display object in the direction of your line of sight, i.e., in front of your face.

[0066] The flowchart in Figure 5 for the case of forced enlargement display is basically the same as the flowchart in Figure 3, except that there is no branch to avoid enlargement display. However, the confirmation of user instruction is changed to a confirmation of whether or not it is a forced enlargement instruction (S103a).

[0067] (Another example of simple enlargement display) Figure 6 shows a modified example of a simple enlarged view.

[0068] As a variation of simple enlargement, as shown in the upper part of Figure 6, a specified sub-region 112 of the display area 111a may be used as the target for enlargement.

[0069] In Figure 6, the specified sub-region 112 is simply enlarged, resulting in the simple enlargement of multiple display objects 302 and 303 within the sub-region 112. Alternatively, a region from the image captured by the rear camera 109 may be extracted, and that extracted sub-image portion may be enlarged. The enlarged objects 304 and 305, which are the enlarged versions of the display objects 302 and 303 that were targeted for enlargement, are both placed in a non-local coordinate system as enlarged region display objects.

[0070] In the example shown in Figure 6, when the processor 101 receives a designation of a partial region 112 and a "display change instruction" from the user as a "display change instruction" (S103), the processor 101 generates an enlarged object 113 which is a simple enlargement of the entire partial region 112. This enlarged object contains enlarged objects 304 and 305 which are simple enlargements of the display objects 302 and 303, respectively. The processor 101 places the enlarged object 113 as an enlarged region display object in a non-local coordinate system (e.g., the world coordinate system) and displays it (S105).

[0071] In addition to the simple magnification shown in Figures 5 and 6, the following are also available as ways to display an enlarged area:

[0072] (Large related display) "Large Related Display" refers to a display mode that switches to another related display object or displays a new related display object. In this case, if the related display object requires a larger display area that exceeds the display area, it will be displayed in an enlarged area. Related display will be explained with reference to Figures 7 to 9. Figures 7, 8, and 9 are examples of related display using menu display as an example.

[0073] In the example in Figure 7, when the user selects an item in the menu object 310 located in the local coordinate system, the "selection instruction" is accepted as a "display change instruction" (S103:YES). The processor 101 reads the submenu 311 associated with the menu object 310 from the display information 34. If the submenu 311 cannot be fully displayed within the display area 111a and it is determined that it needs to be displayed outside the display area 111a (S104:Yes), the "display change instruction" is interpreted as an "enlarged area display instruction," and the submenu 311 is placed in a non-local coordinate system (e.g., the world coordinate system) and displayed (S105). In this case as well, a "forced enlarged area display instruction" that forcibly places the submenu 311 in a non-local coordinate system regardless of its size may also be accepted.

[0074] In the example shown in Figure 8, the submenu 311 is further placed and displayed in a non-local coordinate system (e.g., the world coordinate system) (S105), and a title object 312 indicating that the submenu 311 is placed in a non-local coordinate system is placed in the local coordinate system. That is, in step S105, the processor 101 additionally executes the process of placing the title object 312 in the local coordinate system.

[0075] As a result, if submenu 311 is placed in a non-local coordinate system, depending on the orientation of HMD1, submenu 311 may extend beyond the display area 111a and become invisible. Therefore, the title object 312 is placed in a local coordinate system to notify that submenu 311 is placed in a non-local coordinate system.

[0076] Figure 9 shows an example of how placing the submenu 311 in a non-local coordinate system, similar to Figure 8, can resolve the usability issue where the submenu 311 may extend beyond the display area 111a and become invisible depending on the orientation of the HMD1. In the display area 111a shown in the upper part of Figure 9, approximately the left half of the submenu 311 is displayed in the display area 111a. When the HMD1 is turned to the left, as shown in the display area 111b in the middle part of Figure 9, only the leftmost part beyond the left half of the submenu 311 is displayed. As illustrated in the upper and middle parts of Figure 9, if the remaining portion of the submenu 311 in the display areas 111a and 111b is larger than the size of the residual area 313, the submenu position does not move. Furthermore, when the HMD1 is turned to the left, the submenu 311 is not displayed at all. Therefore, in the display area 111c shown in the lower part of Figure 9, the residual area 313, which is a portion of the submenu 311, is placed in a local coordinate system. This notifies that submenu 311 is located in a non-local coordinate system.

[0077] Figures 10A and 10B are flowcharts showing the processing flow in Figure 9. In Figure 9, steps that overlap with those in Figure 3 are either omitted or given the same step number.

[0078] The processor 101 places and displays the submenu 311 in a non-local coordinate system (e.g., the world coordinate system) (S105) and waits for user instructions (S106). When the processor 101 receives an input instructing it to end the display of the magnified area (S107:YES), it ends the display of the magnified area (S108). If there are no user instructions (S107:NO), it acquires an image from the out camera 109 and sensor outputs from the sensor group 110 (S120), and calculates the position and orientation of the display area 111a in a non-local coordinate system. Then, it compares the display range in the display area 111a where the submenu 311 is displayed with the size of the remaining area 313. The size of the remaining area 313 in the display area 111a is predetermined.

[0079] If the display range within the display area 111a of the submenu 311 is larger than the remaining area 313 (S121: NO), return to step S106 and wait for user instructions.

[0080] If the display range within the display area 111a of submenu 311 is less than or equal to the residual area 313 (S121: YES), the residual area 313 is extracted from submenu 311, placed in the local coordinate system, and displayed. The position of the submenu 311 in the non-local coordinate system at that time is also stored (S122). Display area 111b in Figure 9 shows the point in time when the residual area 313 is extracted. Display area 111c shows the state where the orientation of HMD1 has changed further than that of display area 111b, and only the residual area 313 is displayed.

[0081] The processor 101 waits for user instructions (S106) and, until it receives an instruction to end the display of the enlarged area (S107:NO), acquires images from the out camera 109 and sensor outputs from the sensor group 110 (S123), and monitors whether the position or orientation of the HMD1 has returned to the display start position of the remaining area 313 (S124).

[0082] When the processor 101 determines that the system has recovered (S124: YES), it returns to displaying the submenu 311 from displaying only the residual area 313 (S125). In Figure 9, the display returns to display area 111a. After that, the process returns to step S105.

[0083] If the processor 101 determines that the position or orientation of the HMD1 has not returned to the display start position of the residual area 313 (S124: NO), it returns to step S106 while maintaining the display state of the display area 111c.

[0084] If there is an instruction to end the display of the enlarged area (S107: YES), the display of the enlarged area will end, i.e., the display of submenu 311 will end (S108).

[0085] By using the residual area 313 to display an enlarged area, for objects such as menus that should not completely disappear from the display area, the display position can be adjusted so that at least a minimum portion remains around the periphery of the display area, even if the position or orientation of the HMD1 is changed.

[0086] The above examples illustrate large-scale related displays using menu displays as an example, but large-scale related displays are not limited to menu displays. For example, they could also be the display of new objects associated with hierarchical transitions within an application. Alternatively, they could be the display of new objects associated with the launch of an application located within a local coordinate system.

[0087] Furthermore, for each newly displayed object, the coordinate system to be displayed may be specified in a management table or similar. This specification can be based on the size of the displayed object, or a specific non-local coordinate system can be specified regardless of size, or a local coordinate system can be specified for small objects. In addition, the initial display position of the displayed object may also be specified. When such a display coordinate system is specified, the system will interpret this specification as a user instruction to change the display or an instruction to display an enlarged area, and process it accordingly. Note that the contents of the management table may be made modifiable by the user.

[0088] Furthermore, the same process may be applied not only to displaying new display objects in response to user display change instructions, but also to display objects that are automatically displayed by applications, etc. That is, if a display object that is automatically displayed by an application, etc. can be displayed within the display area of ​​the local coordinate system, it may be placed in the local coordinate system; otherwise, it may be placed in a non-local coordinate system.

[0089] (Display position restriction 1: Evacuation display) When switching to an enlarged area display, if it obstructs the view of other objects, including real objects in the outside world, a "retracted display" may be performed, shifting the display position to a location where it does not obstruct the view. In transparent HMDs, this display mode moves the displayed object to a position where it does not obstruct the view of things in the outside world if it obstructs the view. Figure 11 shows an example of retracted display.

[0090] In the example shown in Figure 11, when the display object 320 is placed in the local coordinate system while the acquaintance 322 is being viewed as an external object, the display object 320 overlaps with the acquaintance 322, as shown in the upper part of Figure 11. Therefore, the processor 101 moves the display object 320 so that it does not overlap with the acquaintance 322 and displays it. As a result of this movement, the area required for displaying the display object 320 goes outside the display area 111a, so the display object 320 is placed in a non-local coordinate system as an enlarged area display object 321.

[0091] Figure 12 is a flowchart showing the processing flow in Figure 11.

[0092] When the processor 101 places the display object 320 in the local coordinate system and displays it (S101), it overlaps with the acquaintance person 322 (Figure 11, upper panel). The processor 101 waits for user instructions in this state (S102). When the user inputs a "retraction instruction" as a display change instruction, the "retraction instruction" becomes a "display enlarged area instruction" (S103: YES).

[0093] The processor 101 calculates the position in a non-local coordinate system where the acquaintance 322 (the person to be evacuated) is visible in the display area 111a based on the image from the rear camera 109 (S130).

[0094] If the processor 101 determines that it is necessary to display outside the display area (S131:YES), it moves the object from the storage target to a non-local coordinate system and displays the enlarged area display object 321 (S132).

[0095] While the processor 101 is waiting for user instructions (S106), if the user inputs a "release save instruction," the "release save instruction" becomes a "end enlarged area display instruction" (S107: YES). The processor 101 changes the position of the display object 320 from a non-local coordinate system to a local coordinate system and ends the enlarged area display (S108). As a result, the display is restored as shown in the lower part of Figure 11.

[0096] The objects to be moved to the side of the display area may be not only external objects, but also other display objects within the display area 111a that obstruct each other's view.

[0097] The above-described "evacuation instruction" and "evacuation release instruction" were explained using an example where the user uses gesture actions as input and the processor 101 recognizes those gesture actions. However, the HMD1's processor 101 may also acquire an image from the out-camera 109, calculate the position of external objects that are visible through the display area 111a, calculate the position of other display objects, determine whether a display object overlaps with external objects or other display objects, and issue an "evacuation instruction" or "evacuation release instruction" according to the result of that determination.

[0098] On the other hand, if the processor 101 determines that it is unnecessary to display outside the display area (S131: NO), it displays the display object while keeping it in the local coordinate system (S109).

[0099] Processor 101 is in the local coordinate system (X L , Y L , Z L With the display object placed in the designated area, the system awaits user instructions (S110).

[0100] If a command to end the display of display object 300 is received (S111:YES), the display is terminated (S112). If no command to end the display is received (S111:NO), the system waits for user instructions (S110).

[0101] Figure 13 shows another example of the evacuation display. In the example in Figure 13, the enlarged area display object 323 is displayed while preemptively avoiding areas where interference is likely to occur (restricted area 400), such as the area directly in front of the user's body. This restricted area 400 is an area defined in a non-local coordinate system.

[0102] Figure 14 is a flowchart showing the processing flow in Figure 13.

[0103] When the user enters "Retract Instruction" as a display change instruction, "Retract Instruction" becomes "Enlarged Area Display Instruction" (S103:YES).

[0104] If the processor 101 determines that the restricted area 400 and the enlarged area display object 323 overlap (S140: YES), it moves the restricted area 400 out of the restricted area and displays the enlarged area display object 323 (S141). After that, it waits for user instructions (S106).

[0105] If the processor 101 determines that the limited area 400 and the enlarged area display object 323 do not overlap (S140: NO), it places the enlarged area display object 323 in a non-local coordinate system and displays it (S142). After that, it waits for user instructions (S106).

[0106] In the case of simple enlargement, the position of the enlargement center may be adjusted so that it does not spread in the forward direction. For example, a predetermined inner area may be used as a reference point from the periphery of the display area 111a, and the enlargement may be set to move away from the restricted area 400 from that point. In addition, before performing the retraction display, the area to be used for display may be indicated in a way that minimizes interference, and the user may be allowed to choose whether or not to perform the retraction display.

[0107] (transparent display) When switching to magnified area display, if it obstructs the visibility of other objects, including real objects in the external environment, you may use "transparency display" to increase the transparency of the displayed objects in the obstructing area.

[0108] Figure 15 shows an example of transparent display. In the example in Figure 15, a region prone to interference (restricted region 400), such as the area directly in front of the user's body, is proactively defined, and the transparency of the display object is increased in the area of ​​the enlarged region display object 323 that overlaps with the restricted region 400.

[0109] In the example in Figure 15, instead of the processor 101 moving the enlarged area display object out of the restricted area 400 and displaying it in a non-local coordinate system in step S141 of Figure 14, it increases the transparency of the part that overlaps with the restricted area 400 and displays the enlarged area display object 323 while keeping its position in the non-local coordinate system the same.

[0110] (Processing of the original display object) There are several ways to handle display objects after the enlarged area display is enabled. Broadly speaking, these can be divided into hiding them or keeping them displayed. Even when keeping them displayed, you can choose an appropriate method, such as displaying them as they are, shifting their display position, displaying them as icons and reducing their size, or displaying them in a lighter color.

[0111] According to the first embodiment, when a display object in a local coordinate system is enlarged and cannot be fully displayed in the local coordinate system, i.e., the display area 111 of the display 103, the enlarged display object is rearranged to a non-local coordinate system as an enlarged area display object. This allows the enlarged area display object to be visible simply by changing the orientation or position of the HMD1, improving usability.

[0112] Furthermore, due to constraints on display area size required for displaying enlarged objects, and limitations on display position due to interference avoidance, etc., if placing the enlarged object in the local coordinate system would prevent the entire object from being visible, it should be placed in a non-local coordinate system and the enlarged area displayed. If the enlarged object fits within the local coordinate system, it should be displayed in the local coordinate system as is. This allows for displaying the object in the local coordinate system as much as possible, improving user convenience.

[0113] <Non-local coordinate system> The non-local coordinate system can be any non-local coordinate system other than the world coordinate system, as long as it is fixed in real space rather than the display area 111 of the HMD1. The non-local coordinate system used may be switched as appropriate by user instruction. The following describes variations of the non-local coordinate system.

[0114] (Chest coordinate system) Figure 16 shows the chest coordinate system (X B , Y B , Z B This is an explanatory diagram.

[0115] The chest coordinate system is a coordinate system fixed to the chest of the user wearing the HMD1. Because display objects can be positioned with the front direction of the chest as the center, the area in which display objects can be positioned can be suitably expanded even when the body orientation changes, without having to strain the neck.

[0116] As a method of fixation, the HMD1's remote controller may be worn around the neck, and the chest coordinate system may be fixed to that remote controller. Alternatively, the HMD1 may capture images of the user's torso, recognize the chest from the images, calculate the distance to the chest, and fix it to the chest.

[0117] (inertial coordinate system) Figure 17 shows the inertial coordinate system (X I , Y I , Z I This is an explanatory diagram.

[0118] An inertial coordinate system is a coordinate system that is fixed to the average position and orientation of the head. It is similar to a chest coordinate system, but while a chest coordinate system is fixed to the chest and does not move, an inertial coordinate system differs in that, for example, when the user's face is turned in a direction that is off-center from the front of the torso due to work requirements, the orientation of the coordinate system is set to follow the direction of the face, i.e., the average orientation of the head. When display objects are placed in an inertial coordinate system, the placement area of ​​the display objects is automatically positioned near the front of the face, improving usability for the user.

[0119] (Direction-fixed non-local coordinate system) Figure 18 shows a direction-fixed nonlocal coordinate system (X V , Y V , Z V This is an explanatory diagram.

[0120] A direction-fixed nonlocal coordinate system is a nonlocal coordinate system different from the world coordinate system, where the vertical direction is aligned with the vertical direction of the real world. For example, the coordinate system is rotated so that the Z-axis direction is always aligned with the vertical direction (Figure 18). The coordinate origin, which serves as the center of rotation, is set to a location near the user. This has the effect of making the display of objects feel more natural, for example, in the case of an HMD1, where the vertical direction of the displayed objects is maintained.

[0121] (Face-fixed non-local coordinate system) Figures 19 and 20 show a plane-fixed nonlocal coordinate system (X P , Y P This is an explanatory diagram.

[0122] Surface-fixed nonlocal coordinate system (X P , Y P This refers to a coordinate system for mobile information terminals (PDAs) that are not worn on the user's body, such as smartphones and tablets, where the distance between the PDAs and the user's viewpoint (eye position) changes. If display objects are placed in the above non-local coordinate system, the size of the display objects on the screen will change when the distance between the smartphone 5 and the user's viewpoint changes. Therefore, a non-local coordinate system consisting of a 2D coordinate system extended from the screen of the flat display mounted on the smartphone 5 or tablet is used.

[0123] Specifically, in the smartphone 5, the position of the smartphone 5 in the non-local coordinate system is changed by the accumulated amount of the component parallel to the screen of the smartphone 5, which represents the amount of movement within the appropriately set three-dimensional non-local coordinate system. At this time, the axis direction of the fixed-surface non-local coordinate system is kept parallel to the axis direction of the local coordinate system of the smartphone 5 (Figure 19). This makes it possible to construct a coordinate system that extends the screen area of ​​the smartphone 5 in a natural way for the user, over a wide area directly in front of the user.

[0124] Furthermore, even if the orientation of smartphone 5 is reversed relative to its casing in real space, the X and Y axes of the fixed-surface nonlocal coordinate system remain parallel to the X and Y axes of the smartphone 5's local coordinate system, respectively (see Figure 20).

[0125] In mobile information terminals such as HMD1, smartphones5, and tablets, one of several non-local coordinate systems may be selected and used in addition to the local coordinate system.

[0126] Additionally, the display screen of a mobile device may be marked or otherwise made clear to the user that the coordinate system being used for display is being used.

[0127] <Second Embodiment> The second embodiment is an embodiment in which an instruction to display an enlarged area and the specification of the coordinate system to be used for display are performed simultaneously.

[0128] Figure 21 is an explanatory diagram of the user interface according to the second embodiment.

[0129] A gesture of spreading fingers is used to instruct zooming in and out, but the number of fingers used also serves to specify the type of non-local coordinate system in which the zoomed-out area display object 331 associated with the display object 330 is placed. For example, three fingers represent the world coordinate system, five fingers represent the chest coordinate system, and so on. Alternatively, a pinch-in operation using two fingers may be configured to convert to a local coordinate system. This makes it possible to control the placement coordinate system with a simple instruction.

[0130] <Third Embodiment> Figure 22 is an explanatory diagram of the user interface according to the third embodiment.

[0131] A display object representing the fixed pin object 500 is prepared and associated with a non-local coordinate system. The operation of pointing the fixed pin object 500 to a display object serves as both the operation of specifying the display object to be converted to an enlarged area display object and the operation of specifying the type of non-local coordinate system to which it will be placed. Display objects that are not pointed to (not pinned) the fixed pin object 500 remain placed in the local coordinate system. When the pin is removed, the system switches to the local coordinate system. Furthermore, the operation of pointing a pin in a non-local coordinate system does not necessarily have to be an instruction to display an enlarged area, but merely a coordinate system switch. This allows for intuitive control of coordinate system switching.

[0132] Furthermore, in the case of a simple enlargement instruction, the object may be enlarged while fixing the position where the pin was inserted.

[0133] According to this embodiment, when displaying objects on the screen of a mobile information terminal such as an HMD1, smartphone5, or tablet, or when displaying related display objects that require a larger display area, the user experience can be improved even with a small display by moving them from a local coordinate system to a non-local coordinate system as needed, thus leading to resource efficiency. Furthermore, since power consumption can be saved by miniaturizing the display, this embodiment is expected to contribute to achieving SDG 7.

[0134] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0135] Furthermore, each of the above configurations may be implemented either partially or entirely in hardware, or through program execution on a processor. Also, the control lines and information lines shown are those deemed necessary for illustrative purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, almost all configurations can be considered interconnected. [Explanation of Symbols]

[0136] 1: HMD 5: Smartphone 10: Cabinet 12: Rear Camera 13: Distance measuring sensor 14: Sensor group 18: Voice input device 19: Audio output device 20:Operator 31: Control Program 32: Application Programs 33: Configuration Information 34:Display information 35: Terminal position and orientation information 100: Controller 101: Processor 101A: Communication Control Unit 101B: Display Control Unit 101C: Data Processing Unit 101D: Data acquisition unit 102: Memory 103: Display 104: Wireless communication device 105: Audio output device 106: Voice input device 107: Operation Input Section 108: Battery 109: Rear Camera 110: Sensor group 111, 111a, 111b, 111c: Display area 112: partial area 113: Enlarged object 114: GPS receiver 115: Distance measuring sensor 141: Accelerometer 142: Gyroscope sensor 143: Geomagnetic sensor 300, 302, 303, 320, 330: Display objects 301, 304, 305: Enlarged objects 310: Menu object 311: Submenu 312: Title object 313 :Residual area 321, 323, 331: Enlarged area display objects 322: person 400: Restricted area 500: Fixed pin object

Claims

1. It is a mobile information terminal, A display that shows objects, An operation input unit that receives operation input for the aforementioned mobile information terminal, The system comprises a processor that controls the display of the aforementioned display, The aforementioned processor, Using a local coordinate system fixed to the mobile device and a non-local coordinate system not fixed to the mobile device, the coordinates for displaying the object are calculated. When a selection operation is performed on a display object displayed in the local coordinate system via the operation input unit, objects related to the display object are placed and displayed in the non-local coordinate system. A portable information terminal characterized by the following features.

2. A portable information terminal, A display that shows objects, Camera and, The system comprises a processor that controls the display of the aforementioned display, The aforementioned processor, Using a local coordinate system fixed to the mobile device and a non-local coordinate system not fixed to the mobile device, the coordinates for displaying the object are calculated. Based on the image captured by the camera, when the user of the mobile information terminal recognizes a gesture action on a display object displayed in the local coordinate system, objects related to the display object are placed and displayed in the non-local coordinate system. A portable information terminal characterized by the following features.

3. A portable information terminal according to claim 1 or claim 2, The object related to the aforementioned display object is an enlarged version of the aforementioned display object. A portable information terminal characterized by the following features.

4. A portable information terminal according to claim 1 or claim 2, The object related to the aforementioned display object is a part of the aforementioned display object. A portable information terminal characterized by the following features.

5. A portable information terminal according to claim 1 or claim 2, The aforementioned display object is a menu object that displays a menu, The object associated with the aforementioned display object is a submenu that represents details for at least one item of the menu object. A portable information terminal characterized by the following features.

6. A portable information terminal according to claim 5, The aforementioned processor, When displaying the submenu, a title object representing the contents of the submenu is placed in the local coordinate system and displayed in the display area of ​​the display. A portable information terminal characterized by the following features.

7. A portable information terminal according to claim 5, The aforementioned processor, If the proportion of the display area of ​​the display occupied by the submenu is less than or equal to a predetermined proportion, the submenu is placed in the local coordinate system. If the proportion of the display area of ​​the display occupied by the submenu is greater than a predetermined proportion, the submenu is placed in the non-local coordinate system. A portable information terminal characterized by the following features.

8. A portable information terminal according to claim 1, Equipped with additional cameras to photograph the outside world, The aforementioned display is a transparent display, The processor processes the physical objects included in the captured image taken by the camera and the display When it is determined that objects overlap, the display objects are moved to the non-local coordinate system. A portable information terminal characterized by the following features.

9. A portable information terminal according to claim 8, When the image captured by the camera no longer includes the physical object, the processor returns the display object to its original display position. A portable information terminal characterized by the following features.

10. A portable information terminal according to claim 1, The aforementioned mobile information terminal further includes a camera that captures images of the user's field of view, The aforementioned display is a transparent display, The aforementioned processor, A restricted area is provided within the display area of ​​the aforementioned display, in which the display object is not to be displayed. The portion of the enlarged area display object that requires enlarged area display and extends beyond the display area of ​​the aforementioned display, which overlaps with the restricted area, is positioned in the non-local coordinate system with increased transparency in that portion. A portable information terminal characterized by the following features.

11. A portable information terminal according to claim 1 or claim 2, The non-local coordinate system is a chest coordinate system consisting of a coordinate system fixed to the chest of the user of the mobile information terminal. A portable information terminal characterized by the following features.

12. A portable information terminal according to claim 1 or claim 2, The aforementioned non-local coordinate system is an inertial coordinate system fixed to the average position and orientation of the user's head of the mobile information terminal. A portable information terminal characterized by the following features.

13. A portable information terminal according to claim 1 or claim 2, The aforementioned non-local coordinate system is a non-local coordinate system different from the world coordinate system, and is a direction-fixed non-local coordinate system in which the vertical direction of the non-local coordinate system coincides with the vertical direction of the real world. A portable information terminal characterized by the following features.

14. A portable information terminal according to claim 1 or claim 2, The aforementioned mobile information terminal is a smartphone or a tablet device, The aforementioned non-local coordinate system is a surface-fixed non-local coordinate system consisting of a two-dimensional coordinate system that extends the screen of a flat-panel display mounted on the smartphone or tablet device. A portable information terminal characterized by the following features.

15. A portable information terminal according to claim 1 or claim 2, The aforementioned processor, A coordinate system fixing object is displayed, which means that the coordinate system of the aforementioned display object is fixed. When the coordinate system fixing object is added to the display object and displayed, the display object is fixed to the coordinate system in which it is located. A portable information terminal characterized by the following features.

16. A portable information terminal according to claim 1 or claim 2, The aforementioned processor, The system accepts user instructions to forcibly place objects related to the aforementioned display object into a non-local coordinate system. A portable information terminal characterized by the following features.

17. An object display method for displaying a display object on a display mounted on a mobile information terminal, A step of calculating the coordinates for displaying the display object using a local coordinate system fixed to the mobile information terminal and a non-local coordinate system not fixed to the mobile information terminal, When a selection operation is performed on a display object displayed in the local coordinate system via the operation input unit of the mobile information terminal, the steps include: arranging and displaying objects related to the display object in the non-local coordinate system; And, execute A method for displaying objects characterized by the following features.

18. An object display method for displaying objects on a display installed in a mobile information terminal, A step of calculating the coordinates for displaying the display object using a local coordinate system fixed to the mobile information terminal and a non-local coordinate system not fixed to the mobile information terminal, Based on the image captured by the camera of the mobile information terminal, the system recognizes a gesture action performed by the user of the mobile information terminal on a display object displayed in the local coordinate system, and then performs the steps of arranging and displaying objects related to the display object in the non-local coordinate system. A method for displaying objects characterized by the following features.

Images