Display terminal and application control method

Abstract

To actualize high operability of a display terminal which has a small display area while maintaining convenience.SOLUTION: A display terminal 200 having a small display area actualizes high operability while maintaining convenience. The display terminal 200 comprises a photography device 214 and further comprises: an image analysis part 223 which analyzes an image 233 photographed by the photography device 214, and uses area information as information on an area corresponding to a real object present in the outside world, on the image 233 to specify an icon associated with the real object by referring to an icon DB 231; and an application startup part 224 which acquires an application associated with the specified icon from an application 232, and then starts it.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display control technique for a display terminal with a small display area. [Background technology]

[0002] An example of a display terminal with a small display area is an HMD (Head Mount Display), which is supported on the user's head and displays an image in front of the user's eyes. Among HMDs, a see-through type that allows both the outside world and the display image to be viewed has a particularly small display area. As a technology for improving the operability of operations using the display image in such an HMD, for example, there is a technology disclosed in Patent Document 1.

[0003] The HMD system disclosed in Patent Document 1 includes a coordinate system setting unit, a segment selection unit, and an image generation unit. The coordinate system setting unit sets a cylindrical coordinate system divided into a plurality of segments in real space. The segment selection unit sets one of the plurality of segments in the cylindrical coordinate system in a selection standby state based on the orientation of the display. When the selection of the segment in the selection standby state is confirmed, the image generation unit generates a display image to be displayed on the display according to the confirmed segment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2014 / 129105 Summary of the Invention [Problem to be solved by the invention]

[0005] In an HMD, just like commonly used information terminals such as personal computers and smartphones, icons are arranged and displayed on the display, and when a selection is made, the application software associated with the selected icon is launched.

[0006] In the technology disclosed in Patent Document 1, a cylindrical coordinate system is set in the real space around the HMD wearer, divided into a plurality of segments. Then, selectable objects (icons) are displayed in each segment, and the icon displayed in the segment in the line of sight of the HMD wearer is considered to be selected. This allows the user (HMD wearer) to give operational instructions without using a handheld operating device for inputting operations, improving operability.

[0007] However, according to the technology disclosed in Patent Document 1, icons of all applications stored in the HMD are displayed, and a desired one is selected from among them. At this time, since segments for displaying each icon are provided in cylindrical coordinates set around the periphery, the user needs to rotate his / her head to select the desired icon. In addition, it is necessary to prepare segments for the number of selectable icons, and when the number of icons is large, the number of segments also increases, and the area of ​​each segment becomes smaller. Accordingly, the size of the icons displayed in each segment also becomes smaller. In addition, since a large area is used to display the icons, the area for viewing the outside world also becomes smaller. Therefore, the more applications are stored, the higher the convenience, but the lower the operability.

[0008] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique for achieving high operability while maintaining convenience in a display terminal having a small display area. [Means for solving the problem]

[0009] The present invention relates to table A display terminal, The display terminal includes a photographing unit, a display unit, and a control unit, and the display terminal has a search mode, and the control unit controls the photographing unit to Acquire an image of the user's field of view Control the Icon shapes corresponding to executable applications to analyze whether or not a is present in the image; The icon graphic but one be When the icon is selected, the application corresponding to the icon will be automatically started. Control to death, The icon graphic ismultiple be In this case, the icon shape is The display unit and launching an application corresponding to the icon graphic selected by the user. Control .

[0011] In addition, the present invention is table An application control method in a display terminal, comprising: The display terminal includes an image capturing unit, a display unit, and a control unit. The display terminal has a search mode. When the search mode is switched to based on a user's instruction, the control unit controls the image capturing unit to Acquires an image of the user's field of vision the display unit controls the image to start automatically, controls the image to analyze whether or not an icon graphic corresponding to an executable application is present in the image, and if there is one icon graphic, controls the image to automatically start the application corresponding to the icon graphic, and if there are a plurality of icon graphics, controls the image to display the icon graphics on the display unit and start the application corresponding to the icon graphic selected by the user. Effect of the Invention

[0012] According to the present invention, it is possible to realize high operability while maintaining convenience in a display terminal having a small display area. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1A is a system configuration diagram of an application control system according to a first embodiment, and FIGS. 1B to 1E are explanatory diagrams for explaining an overview of application control according to the first embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of an HMD according to the first embodiment. [Diagram 3] FIG. 4A is a functional block diagram of an application control unit according to the first embodiment, and FIG. 4B is a functional block diagram of a gaze instruction function of an instruction acceptance display unit according to the first embodiment. [Figure 4] FIG. 2A is an explanatory diagram for explaining an example of an application database according to the first embodiment, and FIG. 2B is an explanatory diagram for explaining an example of an icon database according to the first embodiment. [Diagram 5] 11 is a flowchart of an application control process according to the first embodiment. [Figure 6] 2A to 2C are explanatory diagrams illustrating an application control process according to the first embodiment. [Figure 7] 2A to 2C are explanatory diagrams illustrating an application control process according to the first embodiment. [Figure 8] FIG. 13 is an explanatory diagram for explaining a user instruction method in variant example 1 of the first embodiment, FIG. 13 is an explanatory diagram for explaining an example of icon display in variant example 2 of the first embodiment, and FIG. 13 is an explanatory diagram for explaining a user instruction method when launching an app in variant example 5 of the first embodiment. [Figure 9] 13A to 13C are explanatory diagrams for explaining objects in the fourth modified example of the first embodiment, and FIG. 13D to FIG. 13F are explanatory diagrams for explaining a display example after identifying an icon in the fourth modified example of the first embodiment, respectively. [Figure 10] FIG. 13A is an explanatory diagram for explaining an application database according to a sixth modified example of the first embodiment, and FIG. 13B is an explanatory diagram for explaining an example of a display according to this modified example. [Figure 11] FIG. 23 is an explanatory diagram for explaining an example of a store database according to a seventh modified example of the first embodiment. [Figure 12] 13(a) and 13(b) are explanatory diagrams for explaining an eighth modified example of the first embodiment. [Figure 13] 13(a) to 13(c) are explanatory diagrams illustrating an application control process according to an eighth modified example of the first embodiment. [Figure 14] FIG. 13 is an example of a system configuration diagram of a ninth modified example of the first embodiment. [Figure 15] FIG. 13A is a system configuration diagram of the second embodiment, and FIG. 13B is a hardware configuration diagram of a mobile terminal of the second embodiment. [Figure 16] 13(a) to 13(c) are diagrams illustrating examples of displays on a mobile terminal according to a second embodiment. [Figure 17] FIG. 13A is a system configuration diagram of a third embodiment, and FIG. 13B is a functional block diagram of an HMD of the third embodiment. [Figure 18] 13A to 13C are explanatory diagrams illustrating an application control process according to a third embodiment. [Figure 19] 13A to 13D are explanatory diagrams illustrating an application control process according to a third embodiment. [Figure 20]FIG. 23 is a functional block diagram of an HMD according to a twelfth modified example of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. In the following, in this specification, unless otherwise specified, the same reference numerals are used to refer to components having the same functions, and repeated explanations will be omitted. Note that the present invention is not limited to the embodiments described here.

[0015] <<First embodiment>> In the present embodiment, an application control system is provided that does not display unnecessary icons on a display terminal and starts a desired application, so that a user does not have to search for an icon for starting a desired application from among a large number of icons.

[0016] Hereinafter, in this embodiment (this specification), a figure or symbol that indicates a function of a program and that is selected by a user and causes the program to start when the user instructs execution is called an icon. Furthermore, in this embodiment (this specification), a program that is executed by an execution instruction for an icon is called an application or app.

[0017] In addition, in this embodiment, the display terminal will be described using an HMD (Head Mount Display) that is worn by the user on the head, and is a see-through type HMD that allows the user to see both the outside world and the display.

[0018] 1(a) is a system configuration diagram of an application control system 100 of this embodiment. The application control system 100 of this embodiment includes an HMD 200 and a server 300. The HMD 200 and the server 300 transmit and receive data to and from each other via a network 101.

[0019] The server 300 of this embodiment is a server device installed by a provider of application-related services. In this embodiment, the server 300 analyzes request data transmitted from the HMD 200 and provides related services to the requesting HMD 200. For example, the server 300 holds various applications and provides the applications to the HMD 200 in response to a request from the HMD 200. The server 300 is, for example, an app store or the like.

[0020] In this figure, only one HMD 200 and one server 300 are shown connected to the network 101, but the number of these is not limited. Other devices may be further connected and data may be transmitted and received between them.

[0021] First, an overview of application control by the application control system 100 of this embodiment will be described with reference to Figs. 1(b) to 1(e).

[0022] As shown in FIG. 1(b), when a user 201 wearing an HMD 200 finds a figure that seems to be an icon on a signboard 501 or the like, the user turns on a search mode of the HMD 200. The search mode will be described later. At this time, the user 201 adjusts the image capturing area displayed on the display of the HMD 200 so that the figure is included in the image capturing area. For example, the user 201 adjusts the distance to the figure, the orientation of the user with respect to the figure, and the like.

[0023] When the search mode is turned ON, the HMD 200 acquires an image within the visual field of the user 201. An example of an image 601 acquired here is shown in Fig. 1(c). Here, a taxi stand is shown as an example.

[0024] Then, the HMD 200 analyzes the acquired image 601 and extracts an area (hereinafter, referred to as an object) 602 on the image 601 that corresponds to a figure or the like in the actual field of view. In FIG. 1(d), the dotted rectangular area is the extracted object 602. Details of the extraction will be described later.

[0025] When the HMD 200 extracts the object 602, it determines that the icon specified by the object 602 has been selected, and starts an application stored in association with the specified icon, as shown in Fig. 1(e). Here, an example is shown in which a taxi dispatch application 603 is started. The user 201 selects a desired process from the application and executes it.

[0026] Conventionally, icons of all available applications stored in the HMD 200 are displayed, and the icon required at that time is selected and executed. However, according to this embodiment, an image of a real object within the field of view of the user 201 is identified as an icon, and the application is automatically started. Therefore, the user 201 does not need to search for and select a desired icon from a large number of icons, and can start a related application required on the spot as if an actual figure existing in the surrounding environment were used instead of an icon.

[0027] The HMD 200 of this embodiment that realizes the above processing will be described below.

[0028] [Hardware configuration] 2 is a hardware configuration diagram of the HMD 200 of this embodiment. The HMD 200 of this embodiment basically has the same configuration as a general-purpose computer (information processing device). That is, as shown in this figure, the HMD 200 includes a CPU 211, a RAM 212, a ROM 213, a camera 214, a display 215, an audio interface (I / F) 216, a communication interface (I / F) 217, a sensor 218, and a bus 219 that connects each unit.

[0029] Camera 214, which is an imaging device, includes outward camera 214a and gaze detection camera 214b. Outward camera 214a is attached so as to include the visual field of user 201 of HMD 200 in its imaging range. Gaze detection camera 214b is a camera that detects the gaze direction of user 201. For this reason, gaze detection camera 214b is attached so as to include the iris, pupil, etc. of the eye of user 201 in its imaging range.

[0030] The display 215 displays images acquired by the camera 214 and screen data generated within the HMD 200. The display 215 is configured, for example, with a transmissive liquid crystal device or organic EL device, or an optical scanning device using MEMS (micro electro mechanical systems). However, the device is not limited to these, and any device may be used as long as it can realize a structure that allows a viewer to see through to the other side of the display 215 while displaying an image on the display 215.

[0031] The audio I / F 216 is, for example, a microphone, a speaker, a buzzer, etc. It inputs external sounds and outputs sounds such as sounds created within the HMD 200 and sounds such as voices and music sent via the communication I / F 217. In this embodiment, the audio I / F 216 is not necessarily provided.

[0032] The communication I / F 217 includes an encoding circuit, a decoding circuit, an antenna, etc., and transmits and receives data (data communication) with other devices via the network 101. In this embodiment, the communication I / F 217 is an I / F that connects to the network 101 via an access point or the like (not shown) or via a base station or the like of a mobile telephone communication network (not shown). Via the communication I / F 217, the HMD 200 transmits and receives data with each server 300 connected to the network 101.

[0033] The connection between the HMD 200 and the access point is established by, for example, a wireless communication system such as Wi-Fi (registered trademark) or other communication system. The connection between the HMD 200 and the base station of the mobile telephone communication network is established by, for example, a Wideband Code Division Multiple Access (W-CDMA) (registered trademark) system, a Global System for Mobile communications (GSM) (registered trademark) system, a Long Term Evolution (LTE) system, or other communication system.

[0034] The sensor 218 detects the current position, inclination, speed, and operation by the user 201 of the HMD 200. The sensor 218 includes, for example, a position information acquisition sensor (position information acquisition device) such as a GPS receiver 218a, a gyro sensor 218b, an acceleration sensor 218c, a geomagnetic sensor 218d, a touch sensor 218e, etc. Note that it is not necessary to include all of the sensors 218.

[0035] In addition, when there is no need to particularly distinguish between the RAM 212 and the ROM 213, they are collectively referred to as the storage device 230. In addition, the audio I / F 216 may not be provided.

[0036] [Function block] 3A is a functional block diagram of an application control unit (hereinafter referred to as an application control unit) 220, which is a function related to the above application control process, of the HMD 200 of this embodiment. The application control unit 220 includes an instruction reception display unit 221, a shooting processing unit 222, an image analysis unit 223, an application launch unit 224, and an application acquisition unit 225.

[0037] These units are realized by the CPU 211 loading a program stored in the storage device 230 into a memory such as the RAM 212 and executing it.

[0038] Furthermore, the storage device 230 stores an icon database (icon DB) 231, an application database (application DB) 232, an image 233, and display data 234.

[0039] The application DB 232 is a database that registers an icon and an application that is started by selecting the icon in association with each other. An example of the application DB 232 is shown in FIG.

[0040] As shown in the figure, the application DB 232 registers information on the application associated with each icon. In this embodiment, for each icon ID 232a, which is information for uniquely identifying the graphic or symbol of the icon (hereinafter referred to as an icon graphic or icon), the name of the application (application name) 232b, the application ID 232c, which is information for uniquely identifying the application, and the application storage location 232d, which is information on the location where the application is saved, are registered. The application storage location 232d is an address in the storage device of the HMD 200. Note that, when the application is not held in the storage device of the HMD 200, it may be, for example, a URL that is the location of the server 300 on the network that holds the application.

[0041] The icon DB 231 is a database for registering icons, an example of which is shown in FIG.

[0042] As shown in the figure, each icon includes an icon ID 231a, an icon figure 231b, a type (category) 231c, and a content 231d. The icon ID 231a is information for uniquely identifying the icon. The icon figure 231b is shape information of a figure to be displayed on the display 215 as the icon.

[0043] In the type 231c, a preset classification is registered for each process of the application associated with the icon, such as transportation, payment, facilities, products, shopping, games, music and video distribution, etc. in this embodiment.

[0044] The content 231d is registered with an explanation of the icon graphic 231b and / or an outline of the processing content of the application associated with the icon. The information registered in the content 231d is used when analyzing the meaning of a captured image to identify a related icon, as described below.

[0045] The storage device 230 also stores an image 233 acquired by the image capture processing unit 222. In addition, various data used when the instruction acceptance display unit 221 generates screen data to be displayed on the display 215 is stored as display data 234.

[0046] The instruction acceptance / display unit 221 accepts operation instructions from the user 201. It also generates screen data from the processing results of each unit and causes the display 215 to display the screen data.

[0047] The instruction acceptance display unit 221 displays, for example, various operation instruction buttons (areas) on the display 215 and accepts selection by the user 201. Data of the operation instruction buttons is stored in advance in a storage device.

[0048] An instruction such as a selection by the user 201 is given, for example, by the line of sight. That is, the line of sight direction of the user 201 is detected and used as a substitute for a pointing device. An area on the display 215 that is in the line of sight is determined to be selected.

[0049] Existing technology is used to detect the gaze direction. For example, the gaze direction is detected based on the position of a moving point of the eye of the user 201 relative to a fixed reference point. For example, the inner corner of the eye is used as the fixed reference point. For example, the pupil, the iris, etc. are used as the moving point. These positions are obtained by analyzing an image captured by the gaze detection camera 214b. For example, a predetermined number of blinks may be further added to detect the selection instruction.

[0050] 3(b) shows a functional block diagram of a function for mainly realizing instructions by gaze of the instruction reception display unit 221. As shown in this figure, the instruction reception display unit 221 includes a display control unit 241, a display image generation unit 242, a gaze detection unit 243, and an analysis control unit 244.

[0051] The display image generating unit 242 generates screen data to be displayed on the display 215. The screen data is generated using various data stored in the storage device 230, for example.

[0052] The display control unit 241 causes the display 215 to display the screen data generated by the display image generating unit 242 .

[0053] The gaze detection unit 243 detects the gaze direction from the image captured by the gaze detection camera 214b.

[0054] The analysis control unit 244 analyzes the gaze direction detected by the gaze detection unit 243, identifies an instruction, and outputs a signal to each unit. When the gaze direction is the same for a predetermined period of time, the analysis control unit 244 determines that an image area present on the gaze is selected. In addition, when the gaze direction is the same for a predetermined period of time and the user blinks a predetermined number of times at a predetermined interval, the analysis control unit 244 may determine that the image area is selected.

[0055] The operation instructions include, for example, an instruction to set the processing of the HMD 200 to a search mode (search instruction), a selection instruction to select a desired icon or application from the extracted icon group, an execution instruction to execute the selected application, etc. When the instruction reception display unit 221 receives each instruction, it outputs a selection instruction signal and an execution instruction signal, respectively.

[0056] The search mode is a mode for capturing an image of the actual field of view of the user 201. When the instruction reception display unit 221 receives a search instruction, it outputs an image capture instruction signal to the image capture processing unit 222 as an instruction to acquire an image. The image capture instruction signal is output until an instruction to end the search mode is received.

[0057] The search instruction is received, for example, when the user 201 selects a search instruction button displayed on the display 215. The search instruction button is displayed on the display 215, for example, after the HMD 200 is started up, or after an instruction to start an application control process is received.

[0058] The selection instruction is, for example, an instruction to accept the selection of a desired icon from among a plurality of icons displayed on the display 215. The execution instruction is an instruction to execute an application. An instruction to execute an application associated with the icon selected by the selection instruction is accepted.

[0059] When the photographing processing unit 222 receives a photographing instruction signal from the instruction receiving display unit 221, it photographs the visual field range (actual visual field) of the user 201 at a predetermined time interval. The photographing is performed using the outward camera 214a. The image 233 acquired by photographing is stored in the storage device 230.

[0060] Note that the imaging processing unit 222 may be configured to perform imaging once when the search mode is entered.

[0061] When the image 233 acquired by the photographing processing unit 222 is stored in the storage device 230 , the image analysis unit 223 analyzes the image 233 and extracts an icon from within the image 233 .

[0062] Specifically, the image analysis unit 223 first extracts, as an object on the image, an area corresponding to an actual object (real object) existing in the external world within the image 233. Then, the image analysis unit 223 specifies an icon related to the real object using area information that is information specifying the area of ​​the extracted object.

[0063] The object extraction is performed by, for example, a known contour extraction (edge ​​detection) method. Then, for all the extracted objects, it is determined whether or not they match the icon graphic 231b held in the icon DB 231.

[0064] The determination is performed, for example, by a known matching method based only on shape. Then, among the extracted objects, an object that matches an icon graphic 231b held in the icon DB 231 is determined to be an icon. Then, the matching icon graphic 231b is specified as an icon related to the object.

[0065] Note that the term "match" used here is not limited to a perfect match, but also includes a comparison result with a predetermined degree of similarity or higher.

[0066] The objects extracted by the image analysis unit 223 from the image 233 may be various real objects such as marks displayed on the street, marks posted or displayed in stores, marks printed on product tags, etc. Also, they include various forms such as marks that are painted, marks printed on stickers, marks displayed on display devices such as tablet terminals and guide boards, etc.

[0067] When there are multiple objects determined to be icons, the image analysis unit 223 displays them on the display 215. Then, a selection instruction is received via the instruction receiving display unit 221. At this time, what is displayed on the display 215 may be an icon graphic 231b that matches the object, or the image data itself of the area of ​​the extracted object.

[0068] The icon may be displayed directly on the retina of the eye. Any display method may be used as long as it can display the outside world within the shooting range of the outward camera 214a in a visually observable manner.

[0069] When a selection instruction is received for a displayed icon, the instruction receiving display unit 221 displays the selected icon in a manner different from that of the non-selected icons. For example, the size may be changed, the color may be changed, or the icon may be surrounded by a special frame. Alternatively, only the selected icon may be displayed in a large size and the other icons may be displayed in a smaller size than the selected icon, or only the selected icon may be displayed in color and the other icons may be displayed in grayscale, etc.

[0070] The application launch unit 224 executes the application associated with the selected icon. Specifically, the application launch unit 224 refers to the application DB 232 to identify the application associated with the selected icon. Here, the application launch unit 224 extracts the application storage location 232d registered in association with the icon ID 231a / 232a of the selected icon, and loads the program of the application from that location into the RAM 212 to launch it.

[0071] If the application of the selected icon is not stored in the storage device 230, the application launch unit 224 transmits an application acquisition signal to the application acquisition unit 225. This is the case, for example, when a URL on the network 101 is registered in the application storage location 232d of the selected icon, or when the icon ID 232a of the selected icon is not registered in the application DB 232. The application acquisition signal includes the icon ID 232a. If a URL is registered in the application storage location 232d, the URL is also included.

[0072] When the application acquisition unit 225 receives an application acquisition signal from the application launch unit 224, the application acquisition unit 225 acquires an application associated with the icon ID 232a included in the application acquisition signal from the server 300. When a URL is received, the application acquisition unit 225 accesses the URL and acquires the application.

[0073] At this time, the application acquisition unit 225 transmits an application acquisition request including the icon ID 232a to the server 300, and receives the application provided thereto, that is, downloads the application.

[0074] Furthermore, application acquisition unit 225 stores the provided application in storage device 230, associates the storage location with icon ID 232a, and registers the storage location in application storage location 232d of application DB 232. Then, application launch unit 224 is notified that registration has been completed. Upon receiving the notification, application launch unit 224 launches the application.

[0075] By registering the downloaded application in the application DB 232, the application can be used as is from the next time onwards.

[0076] Note that the selection instruction may not be issued when the number of objects determined to be an icon by the analysis by the image analysis unit 223 is one. In this case, when the image analysis unit 223 determines that the extracted object is an icon, the application launch unit 224 executes the application associated with the icon as is.

[0077] Although not shown, the HMD 200 of this embodiment further includes a control unit for controlling the operation of each piece of hardware. Specifically, the control unit includes a communication control unit for controlling the operation of the communication I / F 217, an outward camera control unit for controlling the operation of the outward camera 214a, and a sensor control unit provided for each sensor 218 for controlling the operation of each sensor 218.

[0078] The communication control unit controls wireless communication functions of the HMD 200, such as WiFi, Bluetooth (registered trademark), and mobile communication systems such as 4G and 5G, and realizes transmission and reception of data with external devices. The sensor control unit outputs signals detected by each sensor.

[0079] [Processing flow] Next, a flow of application control processing by the application control unit 220 of this embodiment will be described. Fig. 5 is a process flow of the application control processing of this embodiment. The following processing is started when a search instruction is accepted.

[0080] The photographing processing unit 222 causes the outward camera 214a to photograph and acquire an image 233 including the actual field of view of the user 201 (step S1101). The acquired image 233 is stored in the storage device 230.

[0081] The image analysis unit 223 analyzes the image 233 and extracts an object (step S1102). Then, it is determined whether the extracted object is an icon, and if it is an icon, it specifies the icon (step S1103). Here, it refers to the icon DB 231 and determines whether an icon figure 231b having a similar shape to the object is registered. That is, if an icon figure 231b similar to the extracted object is registered in the icon DB 231, it determines the object as the similar icon. Then, it specifies the icon by extracting the icon ID 231a registered in association with the icon figure 231b determined as the similar icon. Furthermore, the image analysis unit 223 counts the number of objects whose icons have been specified as the number of icons during the analysis of the image 233.

[0082] The instruction acceptance display unit 221 determines whether the number of icons is 0 (step S1104). If the number of icons is 0, the process ends.

[0083] On the other hand, if the number is not 0, the instruction acceptance display unit 221 determines whether the number of icons is 1 or not (step S1105).

[0084] If the number of icons is 1 (S1105; Yes), the process proceeds to step S1108, which will be described later.

[0085] On the other hand, if the number of icons is not 1 (S1105; No), that is, if the number of icons is more than one, the instruction acceptance display unit 221 displays the icon figures 231b of all the identified icons on the display 215 (step S1106), and then accepts a selection instruction (step S1107).

[0086] The application launch unit 224 refers to the application DB 232 and determines whether or not the application associated with the icon ID 231a / 232a of the selected icon is registered in the HMD 200 (step S1108). Here, if an address in the storage device 230 is stored as the application storage location 232d, it is determined that the application is registered.

[0087] If an application is registered in the HMD 200, the application is started (step S1109), and the process ends.

[0088] On the other hand, if the application is not registered in the HMD 200, the application acquisition unit 225 is instructed to acquire the application. The application acquisition unit 225 accesses the server 300 to search (step S1111), and acquires (downloads) the application associated with the icon ID 232a of the icon from the server 300 (step S1112). Then, the application acquisition unit 225 registers the downloaded application in the application DB 232 in association with the icon ID 232a (step S11BR>P3). Then, the process proceeds to step S1109.

[0089] In this embodiment, during the search mode, the application control unit 220 executes the above process at predetermined time intervals.

[0090] When each process transition occurs, the previous display is erased and a new display is made. This makes it possible to prevent unsightly images from appearing due to unnecessary displays. Of course, this method of displaying the screen can be applied to all screen transitions in this embodiment.

[0091] [Usage example] Next, a specific example of use of the application control system of this embodiment will be described with reference to Fig. 6(a) to Fig. 7(c). Here, an example will be described in which a user 201 wants to start a taxi dispatch application near a taxi stand. The taxi dispatch application is an application for requesting a taxi dispatch, reserving a taxi, and searching for fares.

[0092] 1(b), for example, when the user 201 comes near a taxi stand, the user 201 sees a signboard 501 with the same figure as the icon of the vehicle dispatch application. The user 201 who sees such a signboard 501 points to a search instruction button 711 displayed on the display 215 with his / her line of sight.

[0093] 6(a) shows what the user 201 sees through the display 215 in this case. As described above, the display 215 is a see-through type. Therefore, the user 201 sees an image generated by the HMD 200 superimposed on a real image 510 in the real space. In this figure, the real image in the real space (real environment) is shown by a solid line, and the image generated by the instruction acceptance display unit 221 is shown by a dashed line. In this figure, a search instruction button 711 is an image.

[0094] This is, for example, a station square etc. Within the visual field of the user 201, there are real objects in the real space, such as a bus stop with a bus graphic 511, a train stop guide with a train graphic 512, a signboard with a taxi graphic 513, a person 514, a flower 515, weeds 516, a dog 517, etc.

[0095] When it is accepted that the user 201 has selected the search instruction button 711 by his / her line of sight, the instruction acceptance display unit 221 causes the shooting processing unit 222 to acquire an image. The imaging field of view of the outward camera 214a is set in advance so as to include the actual field of view of the user via the display 215 of the HMD 200. The image 610 acquired at this time is shown in FIG. 6(b).

[0096] The image analysis unit 223 extracts objects from this image 610. Examples of extracted objects are shown in Fig. 6(c). From the image 610 capturing such an environment, a train figure 611, a bus figure 612, a taxi figure 613, a person figure 614, a flower figure 615, a weed figure 616, a dog figure 617, a circle 618, an ellipse 619, and the like are extracted as objects. Then, the image analysis unit 223 refers to the icon DB 231 for each of the extracted objects to determine whether or not it is an icon.

[0097] That is, the image analysis unit 223 compares these objects with the icon figures 231b registered in the icon DB 231, and determines whether there is a match or a similarity. If there is a similarity among the icon figures 231b registered in the icon DB 231, the image analysis unit 223 identifies the extracted object as the icon figure 231b.

[0098] In the above example, the bus graphic 611, the train graphic 612, and the taxi graphic 613 are determined to be icons, and their respective icon IDs 231a are identified.

[0099] Then, the instruction acceptance display unit 221 extracts the icon graphic 231b of each identified icon from the icon DB 231, and generates it as a display icon 712. An example of the generated display icon 712 displayed on the display 215 is shown in Fig. 7(a). At this time, the display area of ​​the display icon 712 is determined in advance. Note that the display area may be changeable by an instruction from the user 201.

[0100] Then, when a selection instruction is received from the user 201, an application associated with the icon is started. In this example, the icon with the shape of a taxi is selected, and a taxi dispatch application associated with the icon is started.

[0101] At this time, the instruction acceptance display unit 221 displays the display icon 712 for which the selection instruction has been accepted in a different form from the other display icons 712. For example, as shown in Fig. 7(b), the display icon 712 is displayed larger than the other display icons 712.

[0102] Thereafter, in accordance with an instruction from the application starting unit 224, the instruction acceptance display unit 221 causes the display 215 to display the initial screen data 713 of the started application, as shown in FIG. 7(c).

[0103] As described above, the HMD 200 of this embodiment analyzes the image 233 captured by the outward camera 214a according to the instruction of the user 201, extracts real objects existing in the external world, and identifies the associated icons for each of them. Then, the HMD 200 starts an application associated with the identified icon. That is, in this embodiment, an application is started by simply capturing an image of a real object similar to the icon figure.

[0104] In this way, according to this embodiment, various objects arranged in the external world are used as if they were icons to launch the corresponding applications. Therefore, according to this embodiment, the application most likely to be used in the environment in which the user 201 is placed can be launched without bothering the user 201. Since there is no process for accepting a selection from a large number of icon displays, it is possible to provide the HMD 200 that is highly convenient and operable without obstructing the field of view of the user 201.

[0105] Furthermore, according to this embodiment, even if multiple icons of applications likely to be used are identified in the environment where the user 201 is placed, only those icons are displayed. In other words, only the icons likely to be used are selectively displayed.

[0106] Generally, in a terminal constructed based on a smartphone, tablet terminal, personal computer, etc., all icons of applications available on the terminal are displayed, and the user 201 must select from among them. For this reason, a large number of icons are displayed on the display 215.

[0107] However, according to this embodiment, only the minimum number of icons are displayed. Therefore, even on a display terminal with a small display area such as the HMD 200, the field of view of the user 201 is less obstructed. In addition, the effort of searching for a desired icon from among a large number of icons can be eliminated, and the usability is also improved. Therefore, the HMD 200 with high visibility, convenience, and operability can be provided.

[0108] Furthermore, in this embodiment, if an application that the user 201 wants to execute is not registered in the HMD 200, the application acquisition unit 225 acquires and registers it from the server 300. This increases the probability that the desired application will be executed.

[0109] <Variation 1> In the above embodiment, the instruction acceptance display unit 221 accepts an instruction from the user 201 by analyzing the line of sight of the user 201. However, the method of accepting an instruction is not limited to this. For example, sensing using the touch sensor 218e may be used. Within the display area of ​​the display 215, the instruction acceptance display unit 221 recognizes and accepts that the area has been selected by cursor movement and selection action by operation on the cursor display, for example, double tap action on the touch sensor 218e.

[0110] 8(a), the instruction acceptance display unit 221 detects the movement of the fingers of the user 201 with the outward camera 214a. When a gesture operation is detected as the movement of the fingers, the instruction acceptance display unit 221 may determine that the display icon 712 is selected. Note that the gesture operation is, for example, an operation such as touching an area where the display icon 712 is displayed, an operation such as clicking, or an operation such as pinching. The touching operation may be an operation such as the finger overlapping the display icon 712 on the line of sight, or an operation such as touching an actual object.

[0111] <Variation 2> In the above embodiment, as shown in Fig. 7(a), the display icon 712 is displayed in a predetermined area of ​​the display 215, but is not limited to this. For example, as shown in Fig. 8(b), the instruction reception display unit 221 may display the display icon 712 as a virtual object at the position of the original real object in the real field of view of the user 201 on the display 215.

[0112] At this time, the display manner of the display icon 712 may be changed between an icon whose corresponding application is stored in the storage device 230 and an icon whose corresponding application is not stored.

[0113] Furthermore, it is not necessary for the icon graphic 231b to be displayed as the display icon 712. A display indicating that the object has been identified as an icon may be displayed in the area of ​​the object on the display 215 that has been identified as an icon. For example, a dotted frame may be displayed in the position of the object.

[0114] <Modification 3> The display of the display icon 712 may be configured to be switchable between display and non-display according to an instruction from the user 201. When the display icon 712 is not displayed, the user 201 may select the area corresponding to the real object as an icon and start the corresponding application. In this case, the instruction acceptance display unit 221 identifies an icon from the object in the image area selected by the user 201 by analyzing an image of the field of view range, and accepts an instruction. Furthermore, if it is determined that the display will hinder the recognition of the outside world, the instruction acceptance display unit 221 may prohibit the display of the display icon 712. In this case, the instruction acceptance display unit 221 acquires an image of the area corresponding to the real field of view before displaying it, and has the image analysis unit 223 analyze it, and determines whether the display of the display icon 712 will hinder the recognition of the real objects in the outside world. Furthermore, according to the analysis result of the image analysis unit 223, the instruction acceptance display unit 221 may adjust the display icon 712 to be displayed in an area where the recognition of the real objects in the outside world is not hindered as much as possible.

[0115] <Variation 4> In the above embodiment, the image analysis unit 223 extracts a figure within the actual visual field of the user 201 as an object from the acquired image 233. However, the object to be extracted is not limited to a figure. For example, it may be a character string displayed on a signboard or the like. Examples of character strings 621, 622, and 623 to be extracted as objects are shown in Figs. 9(a) to 9(c).

[0116] The character string is extracted by an existing technique, for example, a process such as OCR. Then, the image analysis unit 223 analyzes the meaning of the extracted character string. The semantic analysis is performed by an existing technique, for example, a process such as morphological analysis. Then, the analyzed meaning is compared with the contents 231d of the icon DB 231 to determine whether or not there is a corresponding icon. That is, when the meaning obtained by the analysis matches the contents 231d with a predetermined accuracy, the image analysis unit 223 determines that the icon having the contents 231d is a related icon.

[0117] When an icon related to the extracted character string is specified, the image analysis unit 223 may extract the icon graphic 231b of the icon and cause the instruction acceptance display unit 221 to display the icon on the display 215 as the display icon 712. The display position is in the vicinity of the character string in the actual field of view. Examples of display on the display 215 are shown in Fig. 9(d) to Fig. 9(f). Here, Fig. 9(f) is an example in which multiple icons are specified as related icons. In this case, the icon graphic 231b of all the specified icons is displayed.

[0118] By not limiting the objects to be extracted to shapes, it is possible to extract a wider area corresponding to an icon from within the actual visual field of the user 201. This increases the probability that a desired application will be started. Therefore, convenience for the user 201 is improved.

[0119] <Variation 5> In the above embodiment, when the number of icons is one, the corresponding application is automatically started. However, this is not limiting. For example, it may be configured to accept an execution instruction (execution instruction) from user 201 before the application is started. That is, before step S1109, the execution instruction is accepted and application starting unit 224 starts the application.

[0120] 8(c), the instruction acceptance display unit 221 displays, for example, an instruction button 715 for accepting an execution instruction in a predetermined area of ​​the display 215. Then, a selection by gaze on the instruction button 715 is accepted as an execution instruction.

[0121] By receiving an execution instruction from the user 201 before the start-up, the start-up of unnecessary applications can be reduced.

[0122] In the case of automatic startup, the configuration may be such that an instruction (stop instruction) to stop the startup or operation of the application midway can be received from the user 201 after the automatic startup. The stop instruction can be, for example, a gesture operation by the user 201 as described with reference to Fig. 8(a). The gesture operation may be an operation such as waving a hand or shaking a head.

[0123] Furthermore, this gesture operation may be configured to cancel unintended display. For example, the search instruction button 711 may be selected unintendedly due to the line of sight of the user 201. In this case, the display icon 712, initial screen data 713, etc. displayed thereafter are unintended. In such a case, when the instruction acceptance display unit 221 accepts a stop instruction, it stops the application control process and returns to the original screen.

[0124] Also, when an application is automatically started without receiving an execution instruction, a start prohibition may be set in advance depending on the application. Specifically, a flag indicating start prohibition is set in the record of the corresponding application in the application DB 232. When start prohibition is set, the application starting unit 224 does not start the application in the above step S1109, and notifies, for example, the user 201 that start prohibition is set.

[0125] <Variation 6> In the above embodiment, one application is associated with one icon ID 232a in the application DB 232, but this is not limiting. For example, a plurality of applications may be associated with one icon ID 232a.

[0126] For example, FIG. 10(a) shows an example of the application DB 232 in which vehicle dispatch applications of a plurality of taxi companies are associated with an icon of a taxi shape (icon ID 232a is TAXI 01).

[0127] In this case, the application launch unit 224 outputs information on all applications associated with the icon ID 232a of the icon for which the selection instruction has been received to the instruction reception display unit 221. For example, the application name 232b is output. In response to this, the instruction reception display unit 221 generates image data (application icon) 714 capable of identifying the associated application, as shown in FIG. 10(b), and displays it on the display 215.

[0128] The instruction acceptance display unit 221 accepts an instruction of selection by the user 201 for the application icon 714. Then, the application launch unit 224 launches the application.

[0129] When multiple applications are associated with one icon ID 232a, a priority order may be set for the applications, and the application icons 714 may be displayed in descending order of priority. In this case, the priority order is registered in advance in the application DB 232.

[0130] When a priority order is set, the instruction acceptance display unit 221 may determine that the instruction with the highest priority order has been selected without waiting for a selection from the user 201.

[0131] Furthermore, if the corresponding application is not registered in the application DB 232, the application acquisition unit 225 acquires it from the server 300 as described above. At this time, a plurality of applications may be prepared for one icon even on the server 300. In this case, the instruction acceptance display unit 221 may be configured to display information identifying each application on the display 215 before downloading, and to accept an instruction for selection from the user 201.

[0132] When the application acquisition unit 225 acquires an application from the server 300, there may be a case where an application associated with the icon ID 232a to be transmitted does not exist. In such a case, a predetermined number of highly relevant (suitable) applications may be selected in descending order of relevance (suitability) by an AI (Artificial Intelligence) function on the server 300.

[0133] As a result, the application icons 714 of the selected applications may be displayed in order of suitability. Alternatively, the display may be in order of popularity. Then, upon receiving a selection instruction from the user 201, the application acquisition unit 225 downloads the selected application from the server 300.

[0134] <Variation 7> In addition, in cooperation with other functions of the HMD 200, the number of display icons 712 or application icons 714 may be further reduced. In addition, the number of icons to be compared may be reduced to reduce the processing load on the image analysis unit 223. This can increase the processing speed and increase the probability of identifying an icon and the probability of identifying a desired icon.

[0135] For example, icons to be compared in the icon DB 231 are limited depending on the situation of the user 201. In this case, the image analysis unit 223 first extracts an object from the image using the above method. Then, when determining whether it is an icon, it compares only with the icon figure 231b of an icon having the type 231c or the content 231d that matches the situation.

[0136] For example, if the HMD 200 has a schedule management function, the information may be used. For example, if the schedule management function indicates that the user is planning to take a train, the image analysis unit 223 refers to the content 231d or the type 231c, extracts only the icon figures 231b of train-related icons, and performs matching. The train-related icons are, for example, icons whose content 231d is registered as being train-related, or icons whose type 231c is transportation. The image analysis unit 223 acquires schedule information from the schedule management function, and specifies the matching range by semantic analysis.

[0137] Also, for example, if the HMD 200 has a function for storing a processing history, that information may be used. Also, if the HMD 200 has a function for storing a processing history in association with position information at the time of processing, that information may be used. These may be further used in combination with schedule information.

[0138] For example, if the user 201 has a history of making payments with electronic money at a particular store, the image analysis unit 223 uses this information to limit the range of matching. Store location information is used to identify the store. The store location information is acquired by the GPS receiver 218a, which is a location information acquisition device.

[0139] That is, when the current position of the HMD 200 is the store, the image analysis unit 223 compares the extracted object with only the icon figures 231b related to electronic money in the icon DB 231. For example, the icon has information indicating electronic money registered in the content 231d, or the icon has information indicating payment registered in the type 231c.

[0140] Furthermore, if it is recorded that a specific electronic money is used for payment at a specific store, the application launch unit 224 may be configured to directly launch an application related to the specific electronic money when the user 201 turns on the search mode in the store. The application launch unit 224 receives location information and a search instruction via the instruction reception display unit 221.

[0141] Furthermore, when information about the location of the current location is available in addition to the current location information, the range of icons to be extracted may be limited according to this information.

[0142] For example, if the current location is an administrative facility or the like, the image analysis unit 223 searches for administrative service icons, and if the current location is a transportation facility, the image analysis unit 223 searches for only transportation icons.

[0143] As a result, even if a real object similar to an icon is inadvertently photographed, it will not be extracted as an icon. The information on the current location is determined using current location information from the GPS receiver 218a, current location information obtained by the mobile communication function of the communication I / F 217, map information, and the like.

[0144] Also, it may be linked to other databases. For example, as shown in Fig. 11, applications available for each store may be held in advance as a database (store DB 235), and the number of icon graphics 231b to be matched may be reduced by using the database.

[0145] In the store DB 235, the location information and the app ID 232c or icon ID 232a of the available application for each store are registered. In the example of FIG. 11, the case where the icon ID 232a is registered is illustrated. When the current location information acquired by the GPS receiver 218a of the HMD 200 matches the location information of the store, the image analysis unit 223 determines that the user is in the store. Then, the matching range is limited by the available icon ID 232a registered in the store DB 235.

[0146] In addition, in the case where the store DB235 is provided, when it is determined that the user 201 is in a store and the search mode is turned ON, the instruction reception display unit 221 may extract the icon figure 231b of the icon ID 232a registered in the store DB235 from the icon DB231, and display all of them on the display 215, instead of having the image analysis unit 223 analyze the image.

[0147] Furthermore, a priority order of available applications may be set for each store in the store DB 235. In this case, when the search mode is turned ON in the store, the instruction reception display unit 221 may cause the application launch unit 224 to launch the application with the highest priority among the applications registered in the HMD 200.

[0148] In particular, when this modified example is applied to application control processing during payment, etc., icons unrelated to payment are not displayed even if the HMD 200 has application icons corresponding to real objects present in the real field of view. This allows only highly relevant selection candidates to be selected, further improving usability for the user.

[0149] This modification is applicable to a case where a plurality of icons of the same type are displayed and an icon is selected according to a predetermined rule, such as the frequency of past use or a predetermined priority order.

[0150] <Variation 8> Furthermore, when an auxiliary code such as a QR code (registered trademark) is placed near an object corresponding to an icon, auxiliary information provided by the auxiliary code may be used to reduce the processing load on the application control unit 220. For example, an application may be acquired according to the information of the auxiliary code. Note that the following description will be given taking the case where the auxiliary code is a QR code as an example.

[0151] In this embodiment, when the object extracted by the image analysis section 223 is determined to be an icon, the image analysis section 223 further refers to auxiliary information of a QR code in the vicinity of the object in the image 233.

[0152] Note that if there is no QR code in the image, the image analysis unit 223 does not need to perform this process. If there are multiple QR codes in the image, for each object, the QR code that is closest (nearest) in the image is determined to be the QR code of that object.

[0153] Then, the image analysis unit 223 acquires information (called QR information) from the identified QR code, performs processing according to the QR information, and displays the results.

[0154] For example, when a URL is registered in the QR information as a storage location of an associated application, the image analysis unit 223 notifies the application acquisition unit 225 of the URL via the application launch unit 224. The application acquisition unit 225 accesses the URL via the communication I / F 217, acquires the application from the server 300 having the URL, and launches the application. In addition, the QR information itself may be converted into text and displayed on the display 215.

[0155] For example, as shown in Fig. 12(a), assume that a QR code 531 is attached to a signboard 501 with a taxi figure 513 attached thereto. The image analysis unit 223 identifies an area corresponding to the taxi figure 513 as an icon from an image 233 obtained by capturing this field of view. Then, the nearest QR code is identified from the image 233, and its information is acquired and used as described above.

[0156] As shown in FIG. 12(a), the QR code placed on the signboard 501 at the taxi stand records, for example, the URL information of a company providing a taxi dispatch service, the URL information of a taxi dispatch application, location information indicating the designated dispatch location, and the telephone number information of the dispatch center.

[0157] In addition, the QR code placed near the icon whose type 231c is shopping records, for example, the URL information of the online shop application, the URL information containing store information, product information of the store, payment information at the time of checkout, etc.

[0158] In addition, the QR code placed near the bus route image records the URL information of a site that lists bus route information, boarding information, fare information, etc., the URL information of a site that provides real-time operation information, and the URL information of a site that provides transfer information.

[0159] In addition, the QR code placed near the figure showing the toilet records the location of the public toilet, URL information of a site that provides guidance information from the current location to the nearest public toilet, etc. Such QR codes are placed, for example, in large stores, shopping malls, etc.

[0160] In addition to the auxiliary information obtained by the QR code, similarly to the above embodiment, the application launch unit 224 may refer to the application DB 232 and launch an application registered in association with the icon.

[0161] The auxiliary information may also be used in the processing of the launched application.

[0162] In this case, the launched application performs a predetermined process together with or based on the auxiliary information that was read. This allows more complex processes to be automated, such as using the auxiliary information obtained from the QR code as parameters when the application is executed. Therefore, by using the auxiliary code in addition to the processing operation by the application software, it is possible to obtain more desirable processing for the user 201 of the HMD 200.

[0163] 12(b), for example, a tag or the like attached to clothing is provided with a mark 518 and a QR code 531. An application for an online store that handles clothing is associated with this mark 518. Also, information identifying the clothing, such as the manufacturer code, product number, color, and size, is registered as auxiliary information in the QR code 531.

[0164] When the app is launched, it can automatically order the same item of clothing from an online store by importing auxiliary information, and can also search for other items of the same clothing in different colors and sizes.

[0165] Also, for example, as shown in FIG. 13(a), marks of available credit cards, available electronic money, and other available payment methods are usually affixed near the cash register (register) 542 of a store. In such an area, a QR code 531 printed on a receipt 541 or the like may be used. In this case, the QR code 531 records information required for payment, such as the total purchase amount, store information, purchased item information, and payment institution, as auxiliary information. A payment application started by selecting an icon can proceed with the payment process using the auxiliary information recorded in the QR code 531. The QR code may be displayed on a display terminal such as a tablet instead of the receipt 541. Furthermore, marks of available credit cards, available electronic money, and other available payment methods may be displayed on the display terminal.

[0166] The photographing of the visual field range and the photographing of receipt 541 may be performed separately. In this case, application control unit 220 starts an application corresponding to an icon extracted and identified from the previously photographed image of the visual field range of user 201, photographs receipt 541, obtains auxiliary information of QR code 531, and reflects it in the processing of the application.

[0167] In the above description, the image analysis unit 223 is described as functioning as a QR code reader. However, the QR code reader function may be provided separately from the image analysis unit 223. In this case, after identifying the QR code, the image analysis unit 223 instructs the reader function to read the QR code.

[0168] Also, in the above description, the QR code is read automatically. However, the QR code may be read only when an instruction is received from the user 201. In this case, the instruction reception display unit 221 receives an instruction from the user 201 and instructs the image analysis unit 223 or the reader function to read. For example, when the line of sight of the user 201 is in the direction of the QR code on the image displayed on the display 215, it is determined that the QR code has been selected.

[0169] In the above example, the object and the QR code are independent. However, there are also icon-attached QR codes that combine an icon graphic with an auxiliary code (QR code).

[0170] In the case of such an icon-attached QR code, the image analysis unit 223 does not determine whether the extracted object is an icon or not, but determines that it is an icon, and obtains auxiliary information recorded in the QR code. Then, it performs subsequent processing according to the auxiliary information.

[0171] 13(b) shows an example of a payment-related icon-attached QR code 532. In this figure, the QR code portion records that the icon type is a payment-related icon, available payment methods, and so on.

[0172] The image analysis unit 223 acquires this information by analyzing the image. Then, the image analysis unit 223 accesses the icon DB 231, extracts all icon figures whose type 231c is a payment system, and displays them on the display 215 as shown in Fig. 13(c). The following process is the same as in the above embodiment.

[0173] Although the icon graphics of a payment system are used as an example, the present invention is not limited to this and can be applied to various services and processes such as transportation, commerce, and administrative services.

[0174] <Variation 9> In the above embodiment and modified example, the HMD 200 itself includes an icon DB 231, and the image analysis unit 223 determines whether or not an extracted object is an icon by referring to the icon DB 231. Similarly, for applications, the HMD 200 itself includes an app DB 232, and launches the application according to the app storage location 232d of the app DB 232.

[0175] However, the HMD 200 itself does not have to include the icon DB 231 and the application DB 232. Instead of the icon DB 231, the server 300 may include an icon bank 311 in which various icon shapes are registered. Also, the HMD 200 may be configured to always acquire applications from the server 300 without the application DB 232.

[0176] 14, each icon registered in the icon bank 311 includes an icon ID 311a, an icon shape 311b, a type 311c, and details 311d. Furthermore, the icon is registered with URL information of the application corresponding to the icon (corresponding application URL 311e) and related information 311f.

[0177] The type 311c is information classified by icon type, like the icon DB 231, for example, transportation, payment, etc. The icon ID 311a is information for uniquely identifying each icon. The corresponding application URL 311e is access information for an application associated with the icon. The related information is, for example, information on the registration source of the icon, information on the right holder, etc.

[0178] It should be noted that the server 300 in which the icon bank 311 is registered may be a device different from the server 300 in which the application is registered.

[0179] In this case, during the icon identification process in step S1103, the image analysis unit 223 accesses the icon bank 311 of the server 300 via the communication I / F 217, makes an inquiry, and identifies the icon. Then, the image analysis unit 223 downloads the icon ID 311a and icon figure 311b of the identified icon to the HMD 200, displays the icon figure 311b on the display 215, and accepts selection. When downloading, other information associated with the icon (type 311c, content 311d, compatible application URL 311e, etc.) is also downloaded.

[0180] The instruction acceptance display unit 221 notifies information about the selected icon to the application acquisition unit 225. Then, the application acquisition unit 225 accesses the URL according to the corresponding application URL 311e associated with the selected icon, downloads the corresponding application, and starts it.

[0181] When determining whether an extracted object is an icon or not, the processing speed is high if the comparison range is limited to the icon DB 231. On the other hand, by using the icon bank 311, the server 300 is accessed via the network 101, so the overall processing speed is slow, but the possibility of identifying the object as an icon is increased accordingly.

[0182] In this way, by registering various icons and their associated information in the icon bank 311, when an icon is used for the first time, it is possible to recognize the icon and use the associated application, thereby further improving convenience.

[0183] The application itself may also be registered in the icon bank 311 .

[0184] Moreover, the above embodiment and this modified example may be combined. That is, after extracting an object, the image analysis unit 223 first accesses the icon DB 231. Then, if an icon corresponding to the extracted object is not registered in the icon DB 231, the image analysis unit 223 queries the icon bank 311 of the server 300. Then, when the image analysis unit 223 downloads data such as an icon figure from the icon bank 311, it additionally registers the data in the icon DB 231.

[0185] As a result, even if the icon DB 231 is not extensive, if an object similar to an icon in the world can be extracted from the image 233, an application associated with that icon can be started. In other words, for icons that are not widely used, advance registration is not required, further improving convenience.

[0186] If the icon figure 311b that matches the object is not registered in the icon bank 311, the object extracted from the image 233 may be determined to have no similar icon, and a message to that effect may be displayed on the display 215, and the process may be interrupted. Also, the user 201 may be notified of the message.

[0187] By using the icon bank 311, various objects existing in the real world can be registered as icons and used by assigning meanings and categorizing them. Alternatively, even if an object has not been assigned a meaning, it is possible to assign a meaning to it by inferring the meaning from the shape, location, etc. of the object.

[0188] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, the display is performed on the display 215 of the HMD 200, but in this embodiment, a mobile terminal is further linked, and the display is performed on the display of the mobile terminal.

[0189] In this embodiment, the HMD 200 performs wireless communication such as WiFi or Bluetooth (registered trademark) with a mobile terminal held by the user 201, and transmits to the mobile terminal an icon figure specified by the HMD 200, data of the initial screen of an application started, etc. The mobile terminal displays the received icon figure and initial screen on the display of the mobile terminal.

[0190] First, the system configuration of the application control system 100a of this embodiment will be described. Fig. 15(a) is a system configuration diagram of the application control system 100a of this embodiment. The application control system 100a of this embodiment includes an HMD 200, a server 300, and a mobile terminal 400.

[0191] As in the first embodiment, the HMD 200 and the server 300 are connected via a network 101. Furthermore, their configurations and functions are similar to those in the first embodiment.

[0192] However, as described above, in this embodiment, the HMD 200 and the portable terminal 400 transmit and receive data via short-range wireless communication. For this reason, the communication I / F 217 in this embodiment includes a LAN (Local Area Network) communication I / F and a short-range wireless communication I / F.

[0193] The LAN communication I / F is an interface that is connected to a network via an access point (AP) device by wireless connection such as Wi-Fi (registered trademark), and transmits and receives data to and from other devices on the network.

[0194] The near field wireless communication I / F transmits and receives data via wireless communication with other devices equipped with wireless communication devices. For example, it is an I / F that realizes simple information exchange using radio waves between information devices at a distance of several meters to several tens of meters using Bluetooth or the like. Also, the near field wireless communication I / F may be, for example, an I / F for near field wireless communication (NFC (Near Field Communication)) and realize two-way communication between devices equipped with NFC chips at an extremely short distance of several centimeters to approximately one meter. For example, it corresponds to a service that uses a contactless IC chip such as electronic money mounted on the main body of a mobile terminal.

[0195] Furthermore, the instruction acceptance display unit 221 accepts an instruction from the mobile terminal 400 via the communication I / F 217 .

[0196] The mobile terminal 400 is an information processing device having a communication function and a display device, such as a smartphone or a tablet terminal. The mobile terminal 400 may be a wearable device such as a watch. When a wearable device is used, a system can be constructed using only the wearable device in combination with the HMD 200, so that the advantage of not blocking the user's hands is maintained. In this embodiment, the mobile terminal 400 is connected to the HMD 200 via short-range wireless communication. The short-range wireless communication is, for example, WiFi or Bluetooth. The mobile terminal 400 is further connected to a network 101.

[0197] Fig. 15(b) shows a hardware configuration diagram of the mobile terminal 400 of this embodiment. The mobile terminal 400 basically has the same configuration as the HMD 200 shown in Fig. 2. That is, the mobile terminal 400 includes a CPU 411, a RAM 412, a ROM 413, a camera 414 (an outward-facing camera 414a and a gaze detection camera 414b), a display 415, an audio interface (I / F) 416, a communication I / F 417, sensors 418 (a GPS receiver 418a, a gyro sensor 418b, an acceleration sensor 418c, a geomagnetic sensor 418d, and a touch sensor 418e), and a bus 419 connecting each unit. The components with the same names as those of the HMD 200 shown in Fig. 2 have the same functions.

[0198] That is, the communication I / F 417 also includes a LAN communication I / F and a short-range wireless communication I / F, like the communication I / F 217. The communication I / F 417 may further include a telephone network communication I / F for transmitting and receiving calls and data via wireless communication with a base station of a mobile telephone communication network, a priority communication I / F for transmitting and receiving data with other devices near the HMD 200 via a wired connection means such as a USB (Universal Serial Bus), a close-proximity wireless communication I / F for transmitting and receiving data via wireless communication with other devices equipped with a close-proximity wireless communication device, and an infrared communication I / F for realizing infrared communication.

[0199] In this embodiment, the data that is displayed on the display 215 of the HMD 200 in the first embodiment is displayed on a display 415 instead of the display 215 .

[0200] Furthermore, an instruction from the user received via the display 415 is transmitted to the HMD 200 via the communication I / F 417 and then transmitted to the instruction reception display unit 221 via the communication I / F 217 .

[0201] Hereinafter, a specific example of display on the display 415 of this embodiment will be described with reference to Figs. 16(a) to 16(c). For example, when the user 201 approaches a taxi stand, the user selects a search instruction button 711 displayed on the display 215 in the first embodiment. In this embodiment, as shown in Fig. 16(a), a search instruction button 421 is displayed on the display 415. The display is performed by the CPU 411 of the mobile terminal 400.

[0202] When the user 201 selects the search instruction button 421, the CPU 411 accepts the selection. The CPU 411 then generates a signal indicating that the selection has been accepted, and outputs the signal to the HMD 200 via the communication I / F 417. The HMD 200 receives the signal via the communication I / F 217, and passes it to the instruction acceptance display unit 221.

[0203] The instruction reception display unit 221 instructs the image analysis unit 223 to cause the shooting processing unit 222 to acquire an image, as in the first embodiment. Then, the image analysis unit 223 extracts an object and specifies an icon. Then, when the image analysis unit 223 specifies an icon, if the number of specified icons is multiple, the icon figure is transmitted to the mobile terminal 400 via the communication I / F 217.

[0204] 16B, the CPU 411 of the mobile terminal 400 displays the received icon graphic 422 on the display 415. Then, when the CPU 411 accepts a selection from the user, the CPU 411 transmits the accepted selection to the HMD 200 via the communication I / F 417.

[0205] On the HMD 200 side, when the instruction acceptance display unit 221 receives the selection contents via the communication I / F 217, it transmits the selection contents to the application launch unit 224. In response, the application launch unit 224 accesses the application DB 232 in the same manner as in the first embodiment and determines whether or not an application is registered. If the application is registered, the application launch unit 224 launches the application. Then, the instruction acceptance display unit 221 transmits initial screen data to the mobile terminal 400 via the communication I / F 217.

[0206] On the mobile terminal 400 side, the CPU 411 causes the display 415 to display the received initial screen 423, as shown in FIG. 16(c).

[0207] On the other hand, if the application is not registered, the application acquisition unit 225 acquires the application from the server 300, performs processing similar to that of the first embodiment, and then launches the acquired application and transmits its startup screen data.

[0208] According to this embodiment, cooperation is performed with the portable terminal 400 held by the user 201. Therefore, the user 201 can give operational instructions using the portable terminal 400 held by the user 201, which is convenient to use.

[0209] It should be noted that various modified examples of the display mode on the display 215 can also be applied to the display of this embodiment.

[0210] For example, the display of the selected icon graphic may be made different from that of the other non-selected icon graphics, the icon graphic may be superimposed on the acquired image, etc. When the icon graphic is superimposed on the acquired image, the acquired image is also transmitted to the mobile terminal 400.

[0211] Furthermore, various modifications of the first embodiment other than the display mode can also be applied to this embodiment.

[0212] <<Third embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, a beacon is further added to the system configuration of the first or second embodiment. Hereinafter, this embodiment will be described taking the case where a beacon is added to the first embodiment as an example.

[0213] 17A is a system configuration diagram of an application control system 100b of the present embodiment. The application control system 100b of the present embodiment includes an HMD 200, a server 300, and a beacon 800.

[0214] The beacon 800 outputs unique ID information that is set in advance. For example, it constantly transmits a signal (hereinafter, referred to as a beacon signal) conforming to the Bluetooth Low Energy (BLE) standard. This beacon signal is received only by a terminal on which an application for receiving the beacon signal is running.

[0215] The beacon 800 of this embodiment is a transmitter that transmits a beacon signal. The beacon 800 outputs a beacon signal once every few seconds. This beacon signal has an output that can reach an area with a radius of several tens of meters, for example.

[0216] As in the first embodiment, the HMD 200 and the server 300 are connected via a network 101. Furthermore, their configurations and functions are basically the same as in the first embodiment.

[0217] However, the communication I / F 217 of the HDM 200 of this embodiment also functions as a signal receiving unit that receives a beacon signal from a beacon 800. That is, the communication I / F 217 is capable of receiving a signal conforming to the BLE specification.

[0218] Moreover, the HMD 200 goes into search mode when it receives a signal from the beacon 800. To achieve this, the HMD 200 of this embodiment further includes a beacon receiving unit 228 that performs a receiving process of the beacon signal, in addition to the functions of the first embodiment, as shown in FIG.

[0219] When the beacon receiving unit 228 receives a beacon signal from the beacon 800 , it transmits a search instruction to the instruction acceptance display unit 221 .

[0220] As described above, a beacon signal is received only by a terminal equipped with an application (hereinafter, referred to as a beacon receiving application) that performs processing in response to the beacon signal. In this embodiment, a program for this beacon receiving application is registered in the storage device 230. Then, when the HMD 200 is started up, or in accordance with an instruction from the user 201, the CPU 211 loads the beacon receiving application into the RAM 212 and executes it, thereby implementing the communication I / F 217 and the beacon receiving unit 228 that receive the beacon signal. Note that the beacon receiving application is always executed in the background after it is started up.

[0221] Thus, in this embodiment, when the HMD 200 enters an area in which a beacon signal can be received, the HMD 200 receives the beacon signal via the communication I / F 217. The received beacon signal is then transmitted to the beacon receiving unit 228, treated as a search instruction, and transmitted to the instruction acceptance display unit 221.

[0222] Similar to the first embodiment, the instruction acceptance display unit 221 that has received the search instruction causes the photography processing unit 222 to start photography, i.e., acquisition of an image 233. Then, every time an image 233 is acquired, the image analysis unit 223 analyzes the image 233 to extract an object and identify an icon. Then, the application launch unit 224 launches the application of the specified icon.

[0223] As an example, the following description will be given of a case where the beacon 800 is placed in the checkout area of ​​a store, restaurant, etc., as shown in Fig. 18(a). In this case, the beacon 800 is placed near the checkout register (cash register). Usually, marks of usable credit cards or usable IC cards are affixed near the cash register.

[0224] When the user 201 approaches the cash register, the HMD 200 receives a beacon signal from the beacon 800. When the instruction acceptance display unit 221 receives the beacon signal via the communication I / F 217, it causes the shooting processing unit 222 to acquire an image within the field of view. Then, the image analysis unit 223 extracts an object and identifies an icon.

[0225] In the checkout area, the user 201 sees, for example, a scene (real image) 550 as shown in Fig. 18(b) through the transmissive display 215. Specifically, the scene shows stickers of various credit cards and electronic money that can be used at the store affixed near the cash register. Here, by receiving a beacon signal, the photography processing unit 222 captures an image 650 as shown in Fig. 18(c).

[0226] The image analysis unit 223 extracts a group of objects from this image 650. An example of the extracted group of objects 620 is shown in Fig. 19(a). Then, the image analysis unit 223 refers to the icon DB 231 and identifies objects that are icons.

[0227] Then, the instruction acceptance display unit 221 extracts an icon graphic 231b of the object identified as an icon from the icon DB 231, and displays it on the display 215. An example of the display is shown in Fig. 19(b). Note that the outside world view that comes into the field of view of the user 201 through the transparent display 215 is omitted here. The same applies hereinafter.

[0228] The instruction acceptance display unit 221 accepts a selection instruction from the user 201. For example, the instruction acceptance display unit 221 may display the selected icon graphic 231b in a manner different from other icon graphics. For example, as shown in Fig. 19(c), the selected icon graphic is displayed larger than the other icon graphics.

[0229] The application launch unit 224 launches an application associated with the selected icon and registered in the application DB 232. Here, for example, as shown in Fig. 19(d), a screen for making a payment by card indicated by an icon figure is displayed on the display 215.

[0230] If no application is registered in association with the selected icon, the application acquisition unit 225 acquires the application from the server 300, as in the first embodiment.

[0231] In this manner, in this embodiment, a beacon is placed in the vicinity of an object corresponding to an icon of the outside world. Then, instead of turning on the search mode, when a signal from the beacon 800 is received, the HMD 200 starts an application control process.

[0232] Therefore, according to this embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, according to this embodiment, an operation such as turning on the search mode is not required, so that the effort of the user 201 is reduced and convenience is improved.

[0233] <Modification 10> In this embodiment, the beacon signal output from the beacon 800 is basically a signal indicating that an object corresponding to an icon is in the vicinity. However, information other than the location information may be added to the beacon signal transmitted by the beacon 800. For example, information that limits the processing to be performed in the vicinity may be added.

[0234] This makes it possible to limit the types of icons 231c to be checked by the image analysis unit 223. As a result, it is possible to reduce the annoyance of frequently occurring reactions to marks in the outside world.

[0235] For example, in the vicinity of a place involving payment, only an icon related to payment is identified. In this case, when the instruction reception display unit 221 receives a beacon signal, it not only outputs a search instruction but also outputs the beacon signal to the image analysis unit 223.

[0236] The image analysis unit 223 analyzes the beacon signal and identifies the type 231c of the required icon. When identifying an icon corresponding to the extracted object, the image analysis unit 223 checks only icons that match the identified type 231c.

[0237] For example, if the beacon signal is registered to extract only icons whose type 231c is payment, the icon types registered in the icon DB from the left in the example of Fig. 19(b) are payment, payment, payment, payment, and point allocation. In this case, the rightmost icon is not displayed on the display 215.

[0238] In this embodiment, the display destination may be the mobile terminal 400, as in the second embodiment.

[0239] <Modification 11> The HMD 200 in each of the above embodiments and modifications may further include a user authentication function.

[0240] The user authentication function is a security function that authenticates that the user 201 of the HMD 200 is a pre-registered user 201, and instructs the application control process to be performed only if the authentication is successful.

[0241] A functional block diagram of the HMD 200 of this modified example is shown in Fig. 20. The application control unit 220 of this modified example further includes an authentication unit 229 that performs user authentication for authenticating the user 201 of the HMD 200 in addition to the functions of the first embodiment.

[0242] For user authentication, for example, iris authentication can be used. In this case, the user authentication function is realized by, for example, authentication unit 229 and gaze detection camera 214b.

[0243] In iris authentication, an image of the iris of the user 201 is captured in advance and stored as matching data 236 in the ROM 213. When the user 201 wears the HMD 200, the authentication unit 229 acquires an image of the iris of the user 201 with the gaze detection camera 214b and matches it with the pre-stored iris image. If the matching is successful, the authentication unit 229 notifies the instruction acceptance display unit 221 that the user 201 is a legitimate user. If the instruction acceptance display unit 221 receives a notification of successful matching, it displays a search instruction button on the display 215.

[0244] The user authentication is not limited to the above method. For example, the method may be unlocking by tracking the gaze of an authentication pattern, the number of blinks in a predetermined time or blinking at a predetermined time interval, or a fingerprint authentication function. It may also be a combination of these. It may also be another authentication method. The authentication pattern is created by the instruction reception display unit 221, for example.

[0245] As a result, it is possible to prevent spoofing by a third party in, for example, payment processing, etc. Therefore, it is possible to realize the HMD 200 that is highly safe while being convenient and highly operable.

[0246] <Modification 12> In the first embodiment, the third embodiment, and each modification, the HMD 200 is used as the display terminal, but the display terminal is not limited to this. In these examples, the display terminal may be a mobile terminal such as a smartphone or tablet terminal equipped with a camera and a communication function.

[0247] In either case, you can launch an application simply by taking a picture of a real object that resembles the icon shape. This eliminates the need to search for and launch the icon of the application you want to use on the display screen, improving usability.

[0248] In the first and second embodiments, when the search mode is turned on, the image 233 of the outside world is acquired and image analysis is started, but this is not limited to the above. For example, the image 233 of the outside world may be constantly acquired at a predetermined time interval, and the image analysis unit 223 may specify an icon every time an image is acquired.

[0249] In this case, the display icons 712 that can be selected at that time are always displayed on the display 215. The user 201 can simply instruct selection from among the display icons 712 at the timing when he or she wishes to execute an application.

[0250] The present invention is not limited to the above-described embodiments and modifications, but includes various modifications. For example, the above-described embodiments and modifications have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment or modification with the configuration of another embodiment or modification. It is also possible to add the configuration of another embodiment or modification to the configuration of a certain embodiment or modification. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment or modification with another configuration.

[0251] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory unit, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0252] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0253] 100: application control system, 100a: application control system, 100b: application control system, 101: network, 200: HMD, 201: User, 211: CPU, 212: RAM, 213: ROM 214: camera, 214a: outward facing camera, 214b: gaze detection camera, 215: display Play, 216: Audio I / F, 217: Communication I / F, 218: Sensor, 218a: GPS Receiver, 218b: gyro sensor, 218c: acceleration sensor, 218d: geomagnetic sensor 218e: touch sensor, 219: bus, 220: application control unit, 221: instruction reception table display unit, 222: photography processing unit, 223: image analysis unit, 224: application launch unit, 225: application acquisition unit, 228: beacon receiving unit, 229: authentication unit, 230: storage device, 231: Icon DB, 231a: Icon ID, 231b: Icon shape, 231c :Type, 231d:Content, 232:Application DB, 232a:Icon ID, 232b:A Pre-name, 232c: application ID, 232d: application storage location, 233: image, 234: display data, 235: store DB, 236: matching data, 241 : display control unit, 242: display image generating unit, 243: line of sight detecting unit, 244: analysis control unit, 300: Server, 311: Icon Bank, 311a: Icon ID, 311b: Icon 311c:Type, 311d:Content, 311e:Compatible application URL, 311f: Related information, 400: mobile terminal, 411: CPU, 412: RAM, 413: ROM, 414: camera ra, 414a: outward facing camera, 414b: gaze detection camera, 415: display, 417 : communication I / F, 418: sensor, 418a: GPS receiver, 418b: gyro sensor, 418c: acceleration sensor, 418d: geomagnetic sensor, 418e: touch sensor, 419: Bus, 421: search instruction button, 422: icon graphic, 423: initial screen, 501: Signboard, 510: Real image, 511: Bus image, 512: Train image, 513: 514: People, 515: Flowers, 516: Weeds, 517: Dogs, 518: Marks , 531: QR code, 532: QR code with icon, 541: receipt, 550: image, 601: Image, 602: Object, 603: Taxi dispatch application, 610: Image, 611: figure, 612: figure, 613: figure, 614: figure, 615: figure, 616: figure Shape, 617: Figure, 618: Circle, 619: Ellipse, 620: Object group, 621: Character Column, 622: String, 623: String, 650: Image, 711: Search instruction button, 712: Display icon, 713: Initial screen data, 714: App icon, 715: instruction button, 800: beacon

Claims

1. A display terminal, The photography department and A display unit; A control unit, the display terminal has a search mode; The control unit is When the search mode is switched to based on an instruction from a user, the imaging unit is controlled to acquire an image of the user's visual field range; analyzing whether the image has an icon graphic corresponding to an executable application; If there is one icon graphic, control is performed so that an application corresponding to the icon graphic is automatically started; When there are a plurality of the icon figures, the icon figures are displayed on the display unit, and an application corresponding to the icon figure selected by the user is started. Display terminal.

2. 2. The display terminal according to claim 1, The control unit is repeating the process of analyzing and starting an application corresponding to the icon graphic at a predetermined time interval; If the icon graphic is being displayed after the process of starting the application, the display unit is controlled not to display the icon graphic. Display terminal.

3. 2. The display terminal according to claim 1, The display terminal is a head-mounted display. Display terminal.

4. A display terminal according to claim 1, The control unit controls the search mode to end based on an instruction to end the search mode from the user. Display terminal.

5. 2. The display terminal according to claim 1, the display unit displays a search instruction button; The control unit controls to switch to the search mode in response to selection of the search instruction button. Display terminal.

6. A display terminal according to claim 1, When switching to the search mode, the control unit controls the imaging unit to acquire an image of the user's visual field range once. Display terminal.

7. A method for controlling an application in a display terminal, comprising: The display terminal includes: The photography department and A display unit; A control unit, the display terminal has a search mode; The control unit is When the search mode is switched to based on an instruction from a user, the imaging unit is controlled to acquire an image of the user's visual field range; analyzing whether the image has an icon graphic corresponding to an executable application; If there is one icon graphic, control is performed so that an application corresponding to the icon graphic is automatically started; When there are a plurality of the icon figures, the icon figures are displayed on the display unit, and an application corresponding to the icon figure selected by the user is started. A method for controlling an application on a display terminal.

8. A method for controlling an application in a display terminal according to claim 7, comprising: The control unit is repeating the process of analyzing and starting an application corresponding to the icon graphic at a predetermined time interval; If the icon graphic is being displayed after the process of starting the application, the display unit is controlled not to display the icon graphic. A method for controlling an application on a display terminal.

9. A method for controlling an application in a display terminal according to claim 7, comprising: The display terminal is a head-mounted display. A method for controlling an application on a display terminal.

10. A method for controlling an application in a display terminal according to claim 7, comprising: The control unit controls the search mode to end based on an instruction to end the search mode from the user. A method for controlling an application on a display terminal.

11. A method for controlling an application in a display terminal according to claim 7, comprising: the display unit displays a search instruction button; The control unit controls to switch to the search mode in response to selection of the search instruction button. A method for controlling an application on a display terminal.

12. A method for controlling an application in a display terminal according to claim 7, comprising: When switching to the search mode, the control unit controls the imaging unit to acquire an image of the user's visual field range once. A method for controlling an application on a display terminal.

Images