Portable information terminal and object display method

Abstract

To further improve the user friendliness when a display object is displayed while utilizing a local coordinate system and the other coordinate system different from the local coordinate system in conjunction with each other.SOLUTION: A portable information terminal 1 includes a display 103, and a processor 101 that performs a display control on the display. The processor calculates coordinates where a display object is displayed by utilizing, as coordinate systems to display the display object, a local coordinate system fixed to the portable information terminal and at least one or more non-local coordinate system that is not fixed to the portable information terminal, and when displaying an enlarged region display object which is an object relating to the display object being displayed in the local coordinate system and which needs an enlarged region display out of the display region of the display, places the enlarged region display object in the non-local coordinate system.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] A portable information terminal and an object display method that are particularly suitable for displaying virtual objects and are easy to use. [Background technology]

[0002] Conventionally, there is a technology for displaying virtual objects on a head mounted display (HMD). Known coordinate systems used in such virtual object display technology include a world coordinate system and a local coordinate system.

[0003] The world coordinate system is a coordinate system of the real world, and virtual objects placed in the world coordinate system become invisible when the user moves away from that location. On the other hand, since it has the same size as the real world, a large number of virtual objects can be placed.

[0004] On the other hand, the local coordinate system is a coordinate system fixed to the HMD, and its positional relationship with the display mounted on the HMD is fixed. A virtual object arranged in the direction in which the display surface of the display exists as viewed from the user's viewpoint is displayed on the display. If a virtual object is arranged in the local coordinate system within the directional range in which the display surface of the display exists, the display is also fixed to the local coordinate system, so that the virtual object can always be displayed and operated even if the user moves while wearing the HMD. On the other hand, only virtual objects arranged within the above directional range can be displayed, so there is a limit to the number of virtual objects that can be arranged.

[0005] Thus, in the conventional technology that has only two coordinate systems for arranging virtual objects, the world coordinate system and the local coordinate system, there is a problem that it is not possible to arrange a large number of virtual objects that are frequently referred to. Also, there is a problem that the visibility of the outside world is reduced if the virtual objects are forcibly arranged in the direction in which the display screen exists.

[0006] As a technology to solve this problem, Patent Document 1 proposes the following: "A virtual object display device includes a display and a display control device that controls the display of the display, and the display control device detects movement and rotation of the virtual object display device in the real world and defines the position of an inertial coordinate system virtual object using an inertial coordinate system in which a coordinate origin follows the movement of the virtual object device and in which the effective field of view of the display rotates within the coordinate system due to the rotation of the virtual object display device, and a display control unit that displays an inertial coordinate system virtual object in the effective field of view range of the display when the inertial coordinate system virtual object is included in the effective field of view range (summary excerpt)." [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2020 / 157955 Summary of the Invention [Problem to be solved by the invention]

[0008] In Patent Document 1, in addition to a local coordinate system and a world coordinate system, an inertial coordinate system is provided as a coordinate system for arranging virtual objects, thereby increasing the options for object display methods and improving user convenience.

[0009] However, with regard to virtual objects placed in a local coordinate system, there is no consideration of the placement coordinate system when displaying related virtual objects that require a large display area. More specifically, when the size of a virtual object placed in a local coordinate system is changed, there is a problem that the virtual object cannot be viewed outside the display unless a further action, such as scrolling the display, is taken to change the display area.

[0010] The present invention has been made in consideration of the above-mentioned situation, and aims to further improve user convenience when displaying a display object using a local coordinate system and another coordinate system different from the local coordinate system in combination. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention has the configuration described in the claims. One example of the configuration is a mobile information terminal comprising a display that displays a display object, and a processor that performs display control of the display, wherein the processor calculates coordinates for displaying the display object using a local coordinate system fixed to the mobile information terminal and a non-local coordinate system that is not fixed to the mobile information terminal, and arranges the display object and an object related to the display object separately in the local coordinate system and the non-local coordinate system, respectively. Effect of the Invention

[0012] According to the present invention, it is possible to further improve user convenience when displaying a display object by using a local coordinate system and another coordinate system different from the local coordinate system in combination. Objects, configurations, and effects other than those described above will be made clear in the following embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram showing an example of the external configuration of an HMD. [Diagram 2] FIG. 2 is a diagram showing an example of a functional block configuration of an HMD. [Diagram 3] 11 is a flowchart showing the flow of processing for simple enlargement display. [Figure 4] 11A to 11C are explanatory diagrams showing a transition of a display mode in a simple enlarged display. [Diagram 5] 10 is a flowchart showing the flow of a process for forcibly displaying an enlarged area. [Figure 6] FIG. 13 is a diagram showing a modified example of simple enlarged display. [Figure 7] FIG. [Figure 8] FIG. [Figure 9] FIG. [Figure 10A] 10 is a flowchart showing the flow of the process in FIG. 9; [Figure 10B] 10 is a flowchart showing the flow of the process in FIG. 9; [Figure 11] FIG. [Figure 12] 12 is a flowchart showing the flow of the process shown in FIG. 11 . [Figure 13] FIG. 13 is a diagram showing another example of the evacuation display. [Figure 14] 14 is a flowchart showing the flow of the process shown in FIG. 13; [Figure 15] FIG. 13 is a diagram showing an example of transparent display. [Figure 16] Illustration of the chest coordinate system (XB, YB, ZB). [Figure 17] An illustration of the inertial coordinate system (XI, YI, ZI). [Figure 18] An illustration of a direction-fixed non-local coordinate system (XV, YV, ZV). [Figure 19] An illustration of the plane-fixed non-local coordinate system (XP, YP). [Figure 20] An illustration of the plane-fixed non-local coordinate system (XP, YP). [Figure 21] FIG. 11 is an explanatory diagram of a user interface according to the second embodiment. [Figure 22] FIG. 13 is an explanatory diagram of a user interface according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same configurations and processes are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.

[0015] <Hardware configuration> In the following embodiment, a description will be given using an HMD (head mounted display) as a portable information terminal.

[0016] FIG. 1 is a diagram showing an example of the external configuration of the HMD 1. As shown in FIG.

[0017] 1, the HMD 1 is provided with a display 103 including a display area 111 in a glasses-like housing 10. The display 103 is, for example, a transmissive display 103, and a real image of the outside world is transmitted through the display area 111, and an image is superimposed on the real image. The housing 10 is equipped with a controller 100, an outer camera 109, a distance measurement sensor 115, and other sensors (referred to as a sensor group 110 in FIG. 1) other than the distance measurement sensor 115. Note that, although the distance measurement sensor 115 is illustrated separately from the sensor group 110 in FIG. 1, the distance measurement sensor 115 is also a type of sensor, and therefore the sensor group 110 is illustrated as including the distance measurement sensor 115 in FIG. 2, which will be described later.

[0018] The outer camera 109 has, for example, two cameras arranged on the left and right sides of the housing 10, and captures an image by capturing an area including the area in front of the HMD 1. The area including the area in front of the HMD 1 includes the visual recognition area of ​​the user wearing the HMD 1.

[0019] The distance measurement sensor 115 is a sensor that measures the distance between the HMD 1 and an object in the outside world. The distance measurement sensor 115 may use a TOF (Time Of Flight) type sensor, a stereo camera, or other types.

[0020] The sensor group 110 includes a plurality of sensors for detecting the position and orientation state of the HMD 1. On the left and right sides of the housing 10, there are provided an audio input device 106 including a microphone, an audio output device 105 including a speaker and an earphone terminal, and the like.

[0021] The HMD 1 may be provided with an operation device 20 such as a remote controller. In this case, the HMD 1 performs, for example, short-range wireless communication with the operation device 20. A user can input instructions regarding functions of the HMD 1, move a cursor in the display area 111, and the like, by operating the operation device 20 with his or her hand.

[0022] The HMD 1 may communicate with an external device (such as a smartphone or a PC) to cooperate with the HMD 1. For example, the HMD 1 may receive image data of an AR (Augmented Reality) object from an application of the external device.

[0023] The HMD 1 may display a display object in the display area 111. For example, the HMD 1 generates a display object for guiding a user and displays it in the display area 111. When viewed from the user, the display object displayed in the display area 111 becomes an AR object arranged in an augmented reality space added to the real world viewed through the display area 111.

[0024] Fig. 2 is a diagram showing an example of a functional block configuration of the HMD 1 in Fig. 1. In this embodiment, the HMD 1 will be described as an example of a mobile information terminal, but other mobile information terminals such as a smartphone 5 (see Figs. 19 and 20) and a tablet terminal have the same configuration.

[0025] The HMD 1 includes a processor 101, a memory 102, a display 103, a wireless communication device 104, an audio output device 105 including a speaker, etc., an audio input device 106 including a microphone, an operation input unit 107, a battery 108, an outer camera 109, and a group of sensors 110. These elements are connected to each other via a bus or the like.

[0026] The processor 101 is composed of a CPU, a GPU, etc., and constitutes a controller 100 of the HMD 1. The processor 101 executes processing in accordance with a control program 31 and an application program 32 stored in a memory 102, thereby realizing functions of an OS, middleware, applications, etc., and other functions.

[0027] The memory 102 is composed of a ROM, a RAM, etc., and stores various data and information handled by the processor 101, etc. The memory 102 also stores images and detection information acquired by the outer camera 109 as temporary information.

[0028] The outer camera 109 converts light incident through a lens into an electrical signal using an image sensor to obtain an image.

[0029] When the distance measurement sensor 115 is, for example, a TOF (Time Of Flight) sensor, the distance to an object is calculated from the time it takes for light emitted to the outside world to hit the object and return.

[0030] The sensor group 110 includes, for example, an acceleration sensor 141, a gyro sensor (angular velocity sensor) 142, a geomagnetic sensor 143, a GPS receiver 114, and a distance measurement sensor 115. The sensor group 110 uses detection information from these sensors to detect the position, orientation, movement, and other conditions of the HMD 1. The HMD 1 is not limited to these, and may also include an illuminance sensor, a proximity sensor, an air pressure sensor, and the like.

[0031] The display 103 includes a display drive circuit and a display area 111, and displays a display object in the display area 111 based on image data of the display information 34. Note that the display 103 is not limited to a transmissive display, and may be a non-transmissive display or the like.

[0032] The wireless communication device 104 includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Examples of the communication interface include a mobile network, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared, and the like. The wireless communication device 104 performs wireless communication processing between another HMD 1 and an access point. The wireless communication device 104 also performs short-range communication processing with the controller 20.

[0033] The voice input device 106 converts voice input from a microphone into voice data. The voice input device 106 may be equipped with a voice recognition function.

[0034] The audio output device 105 outputs audio from a speaker etc. based on the audio data. The audio output device 105 may have a voice synthesis function.

[0035] The operation input unit 107 is a part that accepts operation inputs to the HMD 1, such as turning the power on or off, adjusting the volume, etc., and is configured with hardware buttons, touch sensors, and the like.

[0036] A battery 108 supplies power to each component.

[0037] The processor 101 includes, as examples of functional block configurations realized by processing, a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D.

[0038] The memory 102 stores a control program 31, an application program 32, setting information 33, display information 34, terminal position and orientation information 35, and the like.

[0039] The control program 31 is a program for realizing the overall control of the HMD 1, including display control.

[0040] The application programs 32 are various programs that the user uses.

[0041] The setting information 33 includes system setting information and user setting information relating to each function.

[0042] The display information 34 includes image data and position coordinate information for displaying a display object in the display area 111 .

[0043] The terminal position and orientation information 35 is information related to the movement and orientation change of the HMD 1 for calculating the position and orientation of the portable information terminal in the non-local coordinate system.

[0044] The communication control unit 101A controls communication processing using the wireless communication device 104 when communicating with other HMDs 1, etc.

[0045] The display control unit 101B uses the display information 34 to control the display of display objects and the like in the display area 111 of the display 103.

[0046] The data processing unit 101C reads and writes the terminal position and orientation information 35, and performs calculations of the position and orientation of the mobile information terminal in the non-local coordinate system.

[0047] The data acquisition unit 101D acquires detection data from various sensors such as the outer camera 109 and the sensor group 110, and generates terminal position and orientation information .

[0048] First Embodiment The first embodiment is an embodiment in which, when a mobile information terminal 1 receives a user's display change instruction for a display object placed in a local coordinate system and a newly displayed display object requires an area larger than the display area of ​​the local coordinate system, the mobile information terminal 1 places the newly displayed display object in a non-local coordinate system and displays the display object.

[0049] Prior to the description of the first embodiment, the terms used in the description of this embodiment will be explained.

[0050] The "local coordinate system" is a coordinate system fixed to the display area 111 of the display 103 of the HMD 1. The user wearing the HMD 1 can see this coordinate system in front of their eyes no matter where they turn their face, as long as the display area 111 is located in front of their eyes.

[0051] A "non-local coordinate system" is a coordinate system that is not fixed to the display area 111 of the display 103. For example, the world coordinate system (X W , Y W , Z W ) The display area of ​​the non-local coordinate system can be changed by changing the orientation and position of the HMD1. In other words, it is a coordinate system in which what is visible changes when the user of the HMD1 turns his or her head. In addition to the world coordinate system, other non-local coordinate systems include a coordinate system that is fixed in front of the user's body from the neck down, and an inertial coordinate system in which the direction in which the user's neck is facing on average is the front. These will be described later.

[0052] An "enlarged area display" is a display that requires an area larger than the display area 111 of the display 103. In other words, an "enlarged area display" is a display that requires an "enlarged area," and does not necessarily mean that the display object is "enlarged." In the following explanation, arranging a display object outside the display area 111 of the display 103 at the same size without "enlarging" it is also referred to as "evacuation."

[0053] The first embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a flowchart showing the flow of processing for simple enlargement display. Fig. 4 is an explanatory diagram showing the transition of the display mode in simple enlargement display. Below, an example will be described in which the display change instruction is a "simple enlargement display instruction" in accordance with the steps in Fig. 3.

[0054] (Simple enlarged view) The simple enlarged display is a display mode in which a display object displayed in a local coordinate system is enlarged and displayed as an enlarged display object with the same content in a local coordinate system or a non-local coordinate system.

[0055] In the initial state, the processor 101 of the HMD 1 displays a display object 300 in the display area 111a as shown in the upper part of Fig. 4 (S101). The coordinate system indicating the display position in the display area 111a of the HMD 1 is a local coordinate system fixed to the HMD 1. The local coordinate system is a three-axis orthogonal coordinate system, with the left and right directions of the display area 111a being Y L axis, the height direction of the display area 111a is Z L The depth direction perpendicular to the display area 111a is the X axis. L The display position of the display area 111a is limited to a certain plane by using two-dimensional coordinates (Y L ,Z L ) It is also possible to arrange a display object three-dimensionally in relation to the local coordinate system. In this case, the display position is expressed as three-dimensional coordinates (X L ,Y L ,Z L ), and the display area visible to the user is a pyramidal area with the user's viewpoint at the apex.

[0056] The processor 101 defines a local coordinate system (X L , Y L , Z L In a state where the display object 300 is arranged in the display area 111a and displayed in the display area 111a, a user instruction is awaited (S102). If there is no display change instruction from the user (S103: NO), the process waits (S102).

[0057] When the user issues a "simple enlargement display instruction" as a display change instruction to display an enlarged display object related to display object 300, and processor 101 accepts the input of the simple enlargement display instruction (S103: Yes), the size of enlarged object 301 obtained by enlarging display object 300 is calculated, and a determination is made as to whether it needs to be displayed outside display area 111a (S104).

[0058] When the processor 101 determines that the enlarged object 301 cannot be displayed within the display area 111a and needs to be displayed outside the display area 111a (S104: Yes), the processor 101 places and displays the enlarged object 301 as an enlarged area display object in a non-local coordinate system (e.g., a world coordinate system) (S105). In the example of FIG. 4, the enlarged object 301, which is a simple enlarged display of the display object 300, cannot be displayed unless it protrudes from the display area 111a. Therefore, the processor 101 places the enlarged object 301 in the non-local coordinate system as an enlarged area display object. To "place in the non-local coordinate system", specifically, the processor 101 calculates the coordinates of the enlarged object 301 in the non-local coordinate system and stores them in the display information 34. By using the terminal position and orientation information 35 and the display information 34, if the enlarged area display object is included in the pyramidal direction position of the display area 111 of the HMD 1, it is displayed on the display 103 of the HMD 1.

[0059] As shown in the lower part of Fig. 4, the part of the enlarged object 301 arranged in the non-local coordinate system that extends beyond the display area 111a cannot be seen as it is. Therefore, when the orientation of the HMD 1 is shifted toward the upper right of the paper surface in Fig. 4, the position in the non-local coordinate system changes from the display area 111a to the display area 111b. A larger part of the enlarged object 301 is displayed in the display area 111b.

[0060] The processor 101 is configured to calculate the coordinate system (X W , Y W , Z W ), the CPU 301 waits for a user instruction (S106).

[0061] When the processor 101 receives an input of an instruction to end the enlarged area display (S107: YES), it ends the enlarged area display (S108). "Ending the enlarged area display" means reducing the enlarged object 301 back to the size of the display object 300 and rearranging it in the local coordinate system. If there is no instruction from the user (S107: NO), it waits (S106).

[0062] On the other hand, if the processor 101 determines that the enlarged object can be displayed within the display area 111a (S104: NO), it displays the enlarged object while keeping it located in the local coordinate system (S109).

[0063] The processor 101 defines a local coordinate system (X L , Y L , Z L ), the system waits for a user instruction (S110).

[0064] When an input of an instruction to end the display of the display object 300 is accepted (S111: YES), the display is ended (S112). If there is no instruction to end the display (S111: NO), the process waits for an instruction from the user (S110).

[0065] (Forced enlarged area display) Forced enlarged area display, which is a modified example of simple enlarged display, will be described. Fig. 5 is a flowchart showing the flow of the forced enlarged area display process. Regardless of the size of the display object after enlargement, the enlarged area display is forcibly used, and the enlarged display object is placed in a non-local coordinate system. This is suitable for enlarging the display object and at the same time, when the user wants to take a good look at the display object in the front direction of the line of sight, i.e., in the front direction of the face.

[0066] The flowchart in Fig. 5 for the forced enlarged region display is basically the same as the flowchart in Fig. 3, except that there is no branch for not displaying the enlarged region. However, confirmation of a user instruction is changed to confirmation of whether or not it is a forced enlarged region instruction (S103a).

[0067] (Another example of simple enlarged display) FIG. 6 is a diagram showing a modified example of simple enlarged display.

[0068] As a modified example of simple enlargement, a specified partial area 112 of a display area 111a may be the target of enlargement display, as shown in the upper part of FIG.

[0069] 6, the specified partial region 112 is simply enlarged, and as a result, the multiple display objects 302, 303 within the partial region 112 are also simply enlarged. A region within the image captured by the outer camera 109 may be cut out, and the cut-out partial image portion may be enlarged. Both enlarged objects 304, 305 obtained by enlarging the display objects 302, 303 to be enlarged are placed in the non-local coordinate system as enlarged region display objects.

[0070] 6, when the processor 101 receives the designation of the partial region 112 and its "enlarged region display instruction" from the user as a "display change instruction" (S103), the processor 101 generates an enlarged object 113 that is a simple enlargement of the entire partial region 112. This includes enlarged objects 304 and 305 that are simple enlargements of the display objects 302 and 303, respectively. The processor 101 places and displays the enlarged object 113 as an enlarged region display object in a non-local coordinate system (e.g., the world coordinate system) (S105).

[0071] In addition to the simple enlarged display shown in Figures 5 and 6, the following are examples of enlarged area displays.

[0072] (Large related display) "Large related display" is a display mode in which a display object is switched to another display object related to the display object, or a new related display object is displayed. In this case, if the related display object requires a large display area exceeding the display area, an enlarged area display is performed. The related display will be described with reference to Figs. 7 to 9. Figs. 7, 8, and 9 are diagrams showing examples of related display using a menu display as an example.

[0073] In the example of FIG. 7, when a user selects an item of menu object 310 arranged in the local coordinate system, the "selection instruction" is accepted as a "display change instruction" (S103: YES). Processor 101 reads submenu 311 associated with menu object 310 from display information 34. If it is determined that submenu 311 cannot be displayed within display area 111a and needs to be displayed outside display area 111a (S104: YES), the "display change instruction" is interpreted as an "enlarged area display instruction" and submenu 311 is arranged and displayed in a non-local coordinate system (e.g., world coordinate system) (S105). In this case, too, a "forced enlarged area display instruction" for forcibly arranging submenu 311 in the non-local coordinate system regardless of its size may be accepted.

[0074] 8, furthermore, submenu 311 is arranged and displayed in a non-local coordinate system (for example, the world coordinate system) (S105), and title object 312 indicating that submenu 311 is arranged in the non-local coordinate system is arranged in the local coordinate system. That is, in step S105, processor 101 additionally executes a process of arranging title object 312 in the local coordinate system.

[0075] As a result, if the submenu 311 is placed in a non-local coordinate system, it may extend beyond the display area 111a and become invisible depending on the orientation of the HMD 1. Therefore, a title object 312 is placed in the local coordinate system to notify that the submenu 311 is placed in a non-local coordinate system.

[0076] FIG. 9 shows an example of solving the inconvenience of the submenu 311 being out of the display area 111a and out of view depending on the orientation of the HMD 1 when the submenu 311 is arranged in the non-local coordinate system as in FIG. 8. In the display area 111a shown in the upper part of FIG. 9, approximately the left half of the submenu 311 is displayed in the display area 111a. When the HMD 1 is turned to the left, only the left side of the left half of the submenu 311 is displayed as in the display area 111b shown in the middle part of FIG. 9. When the partial area of ​​the submenu 311 remaining in the display areas 111a and 111b is equal to or larger than the remaining area 313 as shown in the upper and middle parts of FIG. 9, the submenu position does not move. Furthermore, when the HMD 1 is turned to the left, the submenu 311 is not displayed at all. Therefore, in the display area 111c shown in the lower part of FIG. 9, the remaining area 313, which is a part of the submenu 311, is arranged in the local coordinate system. This informs the user that the submenu 311 is located in a non-local coordinate system.

[0077] Figures 10A and 10B are flow charts showing the flow of the process in Figure 9. Note that in Figure 9, steps that overlap with Figure 3 are omitted or are given the same step numbers.

[0078] The processor 101 places and displays the submenu 311 in a non-local coordinate system (e.g., a world coordinate system) (S105), and waits for a user instruction (S106). When the processor 101 receives an input of an instruction to end the enlarged area display (S107: YES), it ends the enlarged area display (S108). If there is no instruction from the user (S107: NO), it acquires an image from the outer camera 109 and a sensor output from the sensor group 110 (S120), and calculates the position and orientation of the non-local coordinate system of the display area 111a. Then, it compares the display range in which the submenu 311 is displayed in the display area 111a with the size of the remaining area 313. The size of the remaining area 313 that will remain displayed in the display area 111a is determined in advance.

[0079] If the display range of submenu 311 within display area 111a is larger than remaining area 313 (S121: NO), the process returns to step S106 and waits for a user instruction.

[0080] If the display range of the submenu 311 within the display area 111a is equal to or smaller than the residual area 313 (S121: YES), the residual area 313 is cut out from the submenu 311 and arranged and displayed in the local coordinate system. The position of the submenu 311 in the non-local coordinate system at that time is also stored (S122). The display area 111b in Figure 9 shows the point in time when the residual area 313 is cut out. The display area 111c shows a state in which the orientation of the HMD 1 has changed further than in the display area 111b, and only the residual area 313 is displayed.

[0081] The processor 101 waits for a user instruction (S106), and until an instruction to end the enlarged area display is received (S107: NO), it acquires an image from the outer camera 109 and the sensor output from the sensor group 110 (S123), and monitors whether the position or orientation of the HMD 1 has returned to the display start position of the residual area 313 (S124).

[0082] When the processor 101 determines that the display has been restored (S124: YES), the display is restored from only the remaining area 313 to the submenu 311 (S125). In Fig. 9, the display returns to the display area 111a. Then, the process returns to step S105.

[0083] If the processor 101 determines that the position or orientation of the HMD 1 has not returned to the display start position of the remaining area 313 (S124: NO), the processor 101 returns to step S106 while maintaining the display state of the display area 111c.

[0084] If an instruction to end the enlarged area display is given (S107: YES), the enlarged area display is ended, that is, the display of the submenu 311 is ended (S108).

[0085] By displaying an enlarged area using the remaining area 313, in the case of an object such as a menu that should not disappear completely from the display area, the display position can be adjusted so that a minimum portion of the object remains on the periphery of the display area even if the position or orientation of the HMD 1 is changed.

[0086] Although an example of the large related display has been described above using a menu display as an example, the large related display is not limited to a menu display. For example, the large related display may be a display of a new object accompanying a hierarchical transition in an app. Alternatively, the large related display may be a display of a new object accompanying the launch of an app located in a local coordinate system.

[0087] Furthermore, for each newly displayed object, a display coordinate system may be specified in a management table or the like. This specification may be in accordance with the size of the display object, or a non-local coordinate system may be specified regardless of the size, or a local coordinate system may be specified for a small object. Furthermore, an initial display position of the display object may be specified. When such a display coordinate system is specified, the specification is interpreted as a display change instruction by the user or an enlarged area display instruction, and processing is performed. The contents of the management table may be changed by the user.

[0088] Furthermore, in addition to the case where a new display object is displayed in response to a user's display change instruction, the same processing may be performed on a display object that is automatically displayed by an application, etc. In other words, if a display object that is automatically displayed by an application, etc. can be displayed within the display area of ​​the local coordinate system, it may be arranged in the local coordinate system, and if it cannot be displayed, it may be arranged in the non-local coordinate system.

[0089] (Display position restriction 1: Evacuation display) When switching to the enlarged area display, if a display object, including an actual object in the outside world, interferes with the visibility of other objects, a "retraction display" may be performed to shift the display position to a position where it does not interfere. In a see-through HMD, in order to visually recognize things in the outside world, if a displayed object interferes with the visibility of the object, the display object is moved to a position where it does not interfere with the visibility. Fig. 11 is a diagram showing an example of the retraction display.

[0090] In the example of Fig. 11, if the display object 320 is placed in the local coordinate system while an acquaintance 322 is being viewed as an object in the outside world, the display object 320 will overlap the acquaintance 322 as shown in the upper part of Fig. 11. Therefore, the processor 101 moves and displays the display object 320 so that it does not overlap the acquaintance 322. As a result of the movement, the area required to display the display object 320 will go outside the display area 111a, so the display object 320 is placed in the non-local coordinate system as an enlarged area display object 321.

[0091] FIG. 12 is a flowchart showing the flow of the process of FIG.

[0092] When the processor 101 places and displays the display object 320 in the local coordinate system (S101), it is in a state where it is obscured by an acquaintance 322 (upper part of FIG. 11). The processor 101 waits for a user instruction in this state (S102). When the user inputs a "retract instruction" as a display change instruction, the "retract instruction" becomes an "enlarged area display instruction" (S103: YES).

[0093] The processor 101 calculates the position in the non-local coordinate system where the acquaintance 322 (evacuation target) is visible in the display area 111a based on the image from the outer camera 109 (S130).

[0094] When the processor 101 determines that it is necessary to display outside the display area (S131: YES), it evacuates the object from the evacuation target, places it in the non-local coordinate system, and displays the enlarged area display object 321 (S132).

[0095] When the processor 101 is waiting for a user instruction (S106), and the user inputs a "retraction release instruction", the "retraction release instruction" becomes an "enlarged area display end instruction" (S107: YES). The processor 101 changes the location of the display object 320 from the non-local coordinate system to the local coordinate system and ends the enlarged area display (S108). As a result, the display is restored as shown in the lower part of FIG.

[0096] The display objects to be retracted are not limited to objects in the outside world. Retraction display may also be performed in the same manner when display objects within the display area 111a obstruct visibility.

[0097] The above "evacuation instruction" and "evacuation cancellation instruction" have been described using an example in which a gesture action is used as an input operation by the user and the processor 101 recognizes the gesture action, but the processor 101 of the HMD1 may acquire an image from the outer camera 109, calculate the position of an object in the outside world that is visible through the display area 111a, or calculate the position of another display object, determine whether the display object overlaps with an object in the outside world or another display object, and issue an "evacuation instruction" or an "evacuation cancellation instruction" depending on the result of the determination.

[0098] On the other hand, when the processor 101 determines that it is not necessary to display the display object outside the display area (S131: NO), the display object is displayed while being placed in the local coordinate system (S109).

[0099] The processor 101 defines a local coordinate system (X L , Y L , ZL ) and waits for a user instruction (S110).

[0100] When an input of an instruction to end the display of the display object 300 is accepted (S111: YES), the display is ended (S112). If there is no instruction to end the display (S111: NO), the process waits for an instruction from the user (S110).

[0101] Fig. 13 is a diagram showing another example of the retreat display. In the example of Fig. 13, the enlarged area display object 323 is displayed while preemptively avoiding an area (restricted area 400) where obstruction is likely to occur, such as an area in front of the user's body. This restricted area 400 is an area defined in a non-local coordinate system.

[0102] FIG. 14 is a flowchart showing the flow of the process of FIG.

[0103] When the user inputs a "retract instruction" as a display change instruction, the "retract instruction" becomes an "enlarged area display instruction" (S103: YES).

[0104] When the processor 101 determines that the restricted area 400 and the enlarged area display object 323 overlap (S140: YES), the processor 101 moves the enlarged area display object 323 out of the restricted area 400 and displays the object (S141), and then waits for a user instruction (S106).

[0105] If the processor 101 determines that the restricted area 400 and the enlarged area display object 323 do not overlap (S140: NO), the processor 101 places and displays the enlarged area display object 323 in the non-local coordinate system (S142), and then waits for a user instruction (S106).

[0106] In the case of simple enlargement, the position of the center of enlargement may be adjusted so as not to expand in the forward direction. For example, a predetermined inner region from the periphery of the display region 111a may be set as a reference, and the enlargement may be set in a direction away from the restricted region 400. Note that, before performing the retreat display, the area to be used for the display may be displayed in a less obstructive manner, and the user may select whether or not to perform the retreat display.

[0107] (Transparent display) When switching to the enlarged area display, if other objects, including external real objects, are obstructed from viewing, a "transparent display" may be performed in which the transparency of the obstructing parts of the displayed object is increased.

[0108] Fig. 15 is a diagram showing an example of transparent display. In the example of Fig. 15, a region (restricted region 400) where disturbance is likely to occur, such as a region in front of the user's body, is provided as a precaution, and the transparency of the display object is increased in the region of the enlarged region display object 323 that overlaps with the restricted region 400.

[0109] In the example of FIG. 15, instead of the process in step S141 of FIG. 14 in which the processor 101 moves the enlarged region display object out of the restricted region 400 and displays it in the non-local coordinate system, the processor 101 increases the transparency of the portion that overlaps the restricted region 400 and displays the enlarged region display object 323 while keeping its position in the non-local coordinate system unchanged.

[0110] (Processing the original display object) There are also several methods for processing the display object before the enlarged area display after the enlarged area display. Broadly speaking, they can be either hidden or left displayed. If the object is left displayed, an appropriate method can be selected, such as displaying it as is, shifting the display position, iconizing it and displaying it in a reduced size, or displaying it in a lighter color.

[0111] According to the first embodiment, when a display object in a local coordinate system is enlarged and cannot be displayed in the local coordinate system, i.e., the display area 111 of the display 103, the object is rearranged in a non-local coordinate system as an enlarged area display object, which makes it possible to view the enlarged area display object simply by changing the orientation or position of the HMD1, improving usability.

[0112] Furthermore, if the entire enlarged object cannot be seen when placed in the local coordinate system due to constraints such as the size of the display area required to display the enlarged object and restrictions on the display position due to interference avoidance, the object is placed in a non-local coordinate system and displayed in the enlarged area, and if the enlarged object fits in the local coordinate system, it is displayed as is in the local coordinate system, thereby displaying the object in the local coordinate system as much as possible and improving user convenience.

[0113] <Non-local coordinate system> The non-local coordinate system may be a non-local coordinate system other than the world coordinate system as long as it is fixed to the real space and not to the display area 111 of the HMD 1. The non-local coordinate system to be used may be switched by an appropriate instruction from the user. Modifications of the non-local coordinate system will be described below.

[0114] (Chest coordinate system) Figure 16 shows the chest coordinate system (X B , Y B , Z B ) is an explanatory diagram.

[0115] The chest coordinate system is a coordinate system fixed to the chest of a user wearing the HMD 1. Since display objects can be arranged centered on the front direction of the chest, the arrangement area of ​​the display objects can be suitably expanded within a range that does not require the user to turn their neck forcefully even if the body orientation changes.

[0116] As a fixing method, the remote controller of the HMD 1 may be hung around the neck and the chest coordinate system may be fixed to the remote controller. Alternatively, the HMD 1 may capture an image of the user's torso, recognize the chest from the image, and fix the remote controller to the chest by calculating the distance to the chest.

[0117] (inertial coordinate system) Figure 17 shows the inertial coordinate system (X I , Y I , Z I ) is an explanatory diagram.

[0118] An inertial coordinate system is a coordinate system that is fixed to the average position and orientation of the head. It is similar to a chest coordinate system, but the chest coordinate system does not move when fixed to the chest, whereas the inertial coordinate system is different in that, for example, when the user's face is directed in a direction that deviates from the front of the torso due to work, the direction of the coordinate system is set to follow the face direction, i.e., the average head direction. When a display object is placed in an inertial coordinate system, the display object placement area is automatically positioned near the front direction of the face, improving usability for the user.

[0119] (Direction-fixed non-local coordinate system) Figure 18 shows the direction-fixed non-local coordinate system (X V , Y V , Z V ) is an explanatory diagram.

[0120] A direction-fixed non-local coordinate system is a non-local coordinate system that is different from the world coordinate system, in which the vertical direction is aligned with the vertical direction in the real world. For example, the coordinate system is rotated so that the Z-axis direction is always aligned with the vertical direction (Figure 18). The coordinate origin, which is the center of rotation, is set near the user. This has the effect of making it feel more natural, for example in the case of HMD1, if the vertical direction of the displayed object is maintained.

[0121] (Face-fixed non-local coordinate system) Figures 19 and 20 show the plane-fixed non-local coordinate system (X P , Y P ) is an explanatory diagram.

[0122] Plane-fixed non-local coordinate system (X P , Y P) is a coordinate system for a mobile information terminal in which the distance between the mobile information terminal and the user's viewpoint (eyeball position) changes when a mobile information terminal that is not worn on the user's body, such as a smartphone 5 or tablet, is used as the mobile information terminal. When a display object is arranged in the above non-local coordinate system, if the distance from the smartphone 5 to the user's viewpoint changes, the size of the display object on the screen changes, so a non-local coordinate system consisting of a two-dimensional coordinate system extending the screen of the flat display mounted on the smartphone 5 or tablet is used.

[0123] Specifically, the position in the non-local coordinate system of the smartphone 5 is changed by the integrated amount of the components of the movement amount in the appropriately set three-dimensional non-local coordinate system that are parallel to the screen of the smartphone 5. At this time, the axial direction of the plane-fixed non-local coordinate system is kept parallel to the axial direction of the local coordinate system of the smartphone 5 ( FIG. 19 ). This makes it possible to configure a coordinate system that is an extension of the screen of the smartphone 5 in a wide area in front of the user in a natural way for the user.

[0124] Even if the length and width of the smartphone 5 is swapped relative to the body of the smartphone 5 in real space, the X-axis and Y-axis of the surface-fixed non-local coordinate system are parallel to the X-axis and Y-axis of the local coordinate system of the smartphone 5, respectively (see FIG. 20).

[0125] In addition to the local coordinate system, any one of a number of non-local coordinate systems may be selected and used in the HMD 1, the smartphone 5, or a portable information terminal such as a tablet.

[0126] Also, the coordinate system used for display may be displayed on the display screen of the portable information terminal in a manner that allows the user to know the coordinate system by using a mark or the like.

[0127] <Second embodiment> The second embodiment is an embodiment in which an instruction to display an enlarged area and a coordinate system to be used for display are simultaneously specified.

[0128] FIG. 21 is an explanatory diagram of a user interface according to the second embodiment.

[0129] A gesture of spreading the fingers is used to issue a zoom command, but the number of fingers used also serves as an operation to specify the type of non-local coordinate system in which the enlarged area display object 331 related to the display object 330 is placed. For example, three fingers is the world coordinate system, five fingers is the chest coordinate system, etc. Also, a pinch-in operation using two fingers may be configured to convert to a local coordinate system. This makes it possible to control the placement coordinate system with a simple command operation.

[0130] <Third embodiment> FIG. 22 is an explanatory diagram of a user interface according to the third embodiment.

[0131] A display object that resembles a fixed pin object 500 is prepared and associated with a non-local coordinate system. The operation of pointing the fixed pin object 500 at a display object serves both as a designation operation for a display object to be converted into an enlarged area display object and an operation for designating the type of non-local coordinate system in which it will be placed. Display objects that do not point to the fixed pin object 500 (are not pinned) continue to be placed in the local coordinate system. When the pin is removed, the system switches to the local coordinate system. Also, the operation of pointing to a pin in a non-local coordinate system does not have to be a designation for displaying an enlarged area, but merely a switch in the coordinate system. This allows the coordinate system to be switched by an intuitive designation operation.

[0132] Furthermore, in the case of a simple enlargement instruction, the object may be enlarged while fixing the pinned position.

[0133] According to this embodiment, when enlarging a display object displayed on the screen of a mobile information terminal such as the HMD 1, smartphone 5, or tablet, or when displaying a related display object with a larger display area, it is possible to improve usability even on a small display by shifting it from a local coordinate system to a non-local coordinate system as necessary, which leads to resource efficiency. In addition, since power consumption can be reduced by reducing the size of the display, this embodiment is expected to contribute to the achievement of the goals of SDGs 7.

[0134] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0135] In addition, each of the above configurations may be partially or entirely configured as hardware, or may be configured to be realized by executing a program on a processor. In addition, the control lines and information lines are shown as those considered necessary for the explanation, and not all of the control lines and information lines in the product are shown. In reality, it may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0136] 1: HMD 5: Smartphone 10: Housing 12: Rear camera 13: Distance sensor 14: Sensor group 18: Voice input device 19: Audio output device 20:Operator 31: Control program 32: Application program 33: Setting information 34:Display information 35: Terminal position and orientation information 100: Controller 101: Processor 101A: Communication control unit 101B: Display control unit 101C: Data processing section 101D: Data acquisition section 102: Memory 103: Display 104: Wireless communication device 105: Audio output device 106: Voice input device 107: Operation input section 108: Battery 109: Rear camera 110: Sensor group 111, 111a, 111b, 111c: Display area 112: partial area 113: Enlarged object 114: GPS receiver 115: Distance sensor 141: Acceleration sensor 142: Gyro sensor 143: Geomagnetic sensor 300, 302, 303, 320, 330: display objects 301, 304, 305: Expanded objects 310: Menu object 311: Submenu 312: Title object 313 :Residual area 321, 323, 331: Magnification area display object 322: person 400: Restricted area 500: Fixed pin object

Claims

1. A portable information terminal, a display for displaying the display object; A processor that controls display of the display, The processor, Calculating coordinates for displaying the display object using a local coordinate system fixed to the portable information terminal and a non-local coordinate system not fixed to the portable information terminal; The display object and an object related to the display object are arranged separately in the local coordinate system and the non-local coordinate system, respectively. A portable information terminal comprising:

2. 2. The portable information terminal according to claim 1, when displaying both the display object and an object related to the display object in a display area of ​​the display causes a decrease in visibility of the objects, the processor separately arranges the display object and the object related to the display object in the local coordinate system and the non-local coordinate system, respectively; A portable information terminal comprising:

3. 2. The portable information terminal according to claim 1, The object related to the display object is an object obtained by enlarging the display object. A portable information terminal comprising:

4. 2. The portable information terminal according to claim 1, an object related to the display object is a part of the display object; A portable information terminal comprising:

5. 2. The portable information terminal according to claim 1, the display object is a menu object that displays a menu, an object related to the display object is a submenu showing details of at least one item of the menu object; A portable information terminal comprising:

6. 6. The portable information terminal according to claim 5, The processor, when displaying the submenu, a title object indicating the contents of the submenu is placed in the local coordinate system and displayed in a display area of ​​the display. A portable information terminal comprising:

7. 6. The portable information terminal according to claim 5, The processor, if the proportion of the display area of ​​the display occupied by the display of the submenu is equal to or less than a predetermined proportion, locating the submenu in the local coordinate system; if the proportion of the display area of ​​the display occupied by the display of the submenu is greater than a predetermined proportion, locating the submenu in the non-local coordinate system; A portable information terminal comprising:

8. 2. The portable information terminal according to claim 1, Further equipped with a camera for capturing images of the outside world, the display is a transmissive display; When the processor determines that an actual object included in an image captured by the camera overlaps with the display object, the processor moves the display object to the non-local coordinate system. A portable information terminal comprising:

9. 9. The portable information terminal according to claim 8, When the captured image captured by the camera no longer includes the actual object, the processor returns the display object to its original display position. A portable information terminal comprising:

10. 2. The portable information terminal according to claim 1, The mobile information terminal further includes a camera for capturing an image of a visual recognition area of ​​a user of the mobile information terminal, the display is a transmissive display; The processor, providing a restriction area within a display area of ​​the display in which the display object is not displayed, and arranging a portion of an enlarged area display object that protrudes from the display area of ​​the display and requires enlarged area display and overlaps the restriction area in the non-local coordinate system by increasing the transparency of the portion; A portable information terminal comprising:

11. 2. The portable information terminal according to claim 1, The non-local coordinate system is a chest coordinate system formed of a coordinate system fixed to the chest of a user of the mobile information terminal. A portable information terminal comprising:

12. 2. The portable information terminal according to claim 1, The non-local coordinate system is an inertial coordinate system that is fixed to the average position and orientation of the head of the user of the mobile information terminal. A portable information terminal comprising:

13. 2. The portable information terminal according to claim 1, The non-local coordinate system is different from the world coordinate system, and is a direction-fixed non-local coordinate system in which the vertical direction of the non-local coordinate system is aligned with the vertical direction of the real world. A portable information terminal comprising:

14. 2. The portable information terminal according to claim 1, The mobile information terminal is a smartphone or a tablet terminal, The non-local coordinate system is a plane-fixed non-local coordinate system consisting of a two-dimensional coordinate system extending the screen of a flat display mounted on the smartphone or the tablet terminal, A portable information terminal comprising:

15. 2. The portable information terminal according to claim 1, Further equipped with a camera, The processor, Recognizing a gesture movement by a user of the portable information terminal based on an image captured by the camera; receiving, by the gesture action, a designation operation of a coordinate system in which the display object is to be placed and a display conversion operation of the display object; A portable information terminal comprising:

16. 2. The portable information terminal according to claim 1, The processor, displaying a coordinate system fixing object which means fixing a coordinate system of the display object; When the coordinate system fixed object is added to the display object and displayed, the display object is fixed to the coordinate system in which it is placed. A portable information terminal comprising:

17. 2. The portable information terminal according to claim 1, The processor, accepting a user instruction to forcibly place an object related to the displayed object in a non-local coordinate system; A portable information terminal comprising:

18. An object display method for displaying a display object on a display mounted on a mobile information terminal, comprising: Calculating coordinates for displaying the display object using a local coordinate system fixed to the mobile information terminal and a non-local coordinate system not fixed to the mobile information terminal; and placing the display object and an object related to the display object in the local coordinate system and the non-local coordinate system, respectively.

13. A method for displaying objects comprising:

Images