Display terminal and control method for the display terminal

Abstract

To provide a display terminal having a small display area and achieving high operability while maintaining convenience.SOLUTION: High operability while maintaining convenience is achieved in a display terminal having a small display area. A display terminal 200 having a photographing device 214 is provided with an image analysis unit 223 for analyzing an image 233 photographed by the photographing device 214, and specifying an icon related to an actual object by referring to an icon DB 231 using area information, which is information of an area on the image 233 and corresponding to the actual object existing in the outside world, and an application starting unit 224 for acquiring an application associated with the specified icon from an application 232 and starts it.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to display control technology in a display terminal with a small display area.

Background Art

[0002] As a display terminal with a small display area, for example, there is an HMD (Head Mount Display) that is supported on a user's head and displays an image in front of the user's eyes. lay). Among HMDs, the transmissive (see-through type) that enables visual recognition of both the outside world and the display is particularly small in display area. In such an HMD, as a technology for improving the operability of operations performed using display display, for example, there is the technology disclosed in Patent Document 1. display, for example, there is the technology disclosed in Patent Document 1.

[0003] The HMD system disclosed in Patent Document 1 includes a coordinate system setting unit, a segment selection unit, and an image generation unit. The coordinate system setting unit sets a cylindrical coordinate system divided into a plurality of segments on the real space. The segment selection unit sets any one of the plurality of segments in a selection standby state based on the direction of the display in the cylindrical coordinate system. When the selection of the segment in the selection standby state is determined, the image generation unit generates a display image to be displayed on the display according to the determined segment.

[0004]

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the case of HMDs, as with commonly available information terminals such as personal computers and smartphones... In this way, icons are placed and displayed on the screen, and selections are accepted. The application software associated with the selected icon will be launched.

[0006] The technology disclosed in Patent Document 1 divides the real space around the HMD wearer into multiple segments. Set the defined cylindrical coordinates. Then, select an object (eye) for each segment. The console is displayed, and the icon shown in the segment of the HMD wearer's line of sight is selected. This is assumed. As a result, the user (HMD wearer) can perform operations by holding the device in their hand and inputting commands. Operational instructions can be given without using any equipment, improving usability.

[0007] However, according to the technology disclosed in Patent Document 1, all applications stored in the HMD A selection of icons will appear, and you can choose the desired one from among them. At this time, the surrounding settings A segment is provided to display each icon in the cylindrical coordinates, so the desired icon can be displayed. To make a selection, the user needs to think carefully. Also, the number of selectable icons... Therefore, you need to prepare segments, and if there are many icons, the number of segments will also increase. As a result, the area of ​​each segment becomes smaller. Consequently, the icons displayed in each segment The size of the icons will also be smaller. In addition, a large area will be used to display the icons, so the outside world will be less visible. The visible area also becomes smaller. Therefore, the more applications that are saved, the smaller the area becomes. However, while convenience increases, usability decreases.

[0008] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technology that realizes high operability while maintaining convenience in a display terminal having a small display area. While maintaining convenience, it aims to provide a technology that realizes high operability.

Means for Solving the Problems

[0009] The present invention is a display terminal including a photographing unit, a display unit, a control unit, a storage device, and a communication device. The aforementioned The display terminal has a search mode. When the control unit switches to the search mode based on a user's instruction, the control unit controls the photographing unit to acquire an image within the user's field of view, and controls to analyze whether an icon figure corresponding to an executable application is present in the image. The aforementioned storage device When there is no The aforementioned application corresponding to the icon figure, it controls to acquire an application via the communication device. The aforementioned If the storage device does not contain the icon shape corresponding to the application, the device is controlled to acquire the icon shape via the communication device.

[0011] The present invention is table a Control method display terminal When the display terminal switches to search mode based on user instructions, it performs the following steps: acquire an image of the user's field of view; analyze whether an icon shape corresponding to an executable application is present in the image; control the display terminal to acquire the application if the application corresponding to the icon shape is not present in the display terminal; and control the display terminal to acquire the icon shape if the icon shape corresponding to the application is not present in the display terminal.

Advantages of the Invention

[0012] According to the present invention, in a display terminal having a small display area, high operability can be realized while maintaining convenience. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments. clarified.

Brief Description of the Drawings

[0013] [Figure 1] (a) is a system configuration diagram of an application control system according to the first embodiment, and (b) to (e) are explanatory diagrams for explaining the outline of application control according to the first embodiment. [Figure 2] It is a hardware configuration diagram of the HMD according to the first embodiment. [Figure 3](a) is a functional block diagram of the application control unit of the first embodiment, and (b) is a functional block diagram of the line-of-sight instruction function of the instruction reception display unit of the first embodiment. [Figure 4] (a) is an explanatory diagram for explaining an example of the application database of the first embodiment, and (b) is an explanatory diagram for explaining an example of the icon database of the first embodiment. [Figure 5] It is a flowchart of the application control process of the first embodiment. [Figure 6] (a) to (c) are explanatory diagrams for explaining the application control process of the first embodiment. [Figure 7] (a) to (c) are explanatory diagrams for explaining the application control process of the first embodiment. [Figure 8] (a) is the instruction method by the user of Variation 1 of the first embodiment, (b) is an example of icon display of Variation 2 of the first embodiment, and (c) is the instruction method at the time of starting the application of Variation 5 of the first embodiment, respectively explanatory diagrams for explaining. [Figure 9] (a) to (c) are explanatory diagrams for explaining the objects of Variation 4 of the first embodiment, and (d) to (f) are explanatory diagrams for explaining the display examples after icon identification of Variation 4 of the first embodiment, respectively. [Figure 10] (a) is the application database of Variation 6 of the first embodiment, and (b) is the display example of the display of this variation, respectively explanatory diagrams for explaining. [Figure 11] It is an explanatory diagram for explaining an example of the store database of Variation 7 of the first embodiment. [Figure 12] (a) and (b) are explanatory diagrams for explaining Variation 8 of the first embodiment, respectively. [Figure 13] (a) to (c) are explanatory diagrams for explaining the application control process of Variation 8 of the first embodiment. [Figure 14] It is an example of the system configuration diagram of Variation 9 of the first embodiment. [Figure 15](a) is a system configuration diagram of the second embodiment, and (b) is a hardware configuration diagram of the mobile terminal of the second embodiment. [Figure 16] (a) to (c) are explanatory diagrams illustrating an example of display in a mobile terminal according to the second embodiment. [Figure 17] (a) is a system configuration diagram of the third embodiment, and (b) is a functional block diagram of the HMD of the third embodiment. [Figure 18] (a) to (c) are explanatory diagrams illustrating the application control process of the third embodiment. [Figure 19] (a) to (d) are explanatory diagrams illustrating the application control process of the third embodiment. [Figure 20] This is a functional block diagram of an HMD representing a modified example 12 of the present invention. [Modes for carrying out the invention]

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Items with the same function are assigned the same designation unless otherwise specified, and repeated explanations are omitted. The present invention is not limited to the embodiments described herein.

[0015] <<First Embodiment>> In this embodiment, the display terminal does not display unnecessary icons and the desired application is displayed. It provides an application control system that launches the application. Users can access multiple icons. You don't have to search through the list of icons to find the one that will launch your desired application.

[0016] Hereinafter, in this embodiment (specified specification), graphics or symbols that indicate the function of the program are used. The program is launched by the user selecting and instructing it to run. The symbols are called icons. Furthermore, in this embodiment (specified specification), the actual A program executed by line directives is called an application or app.

[0017] Furthermore, in this embodiment, the display terminal is an HMD (Head-Mounted Display) worn by the user on their head. A head-mounted display (HMD) is a device that connects the external world to the digital world. Let's take a transparent (see-through) HMD, which allows both the display and the HMD to be visible, as an example. do.

[0018] Figure 1(a) is a system configuration diagram of the application control system 100 of this embodiment. Yes. The application control system 100 of this embodiment includes an HMD 200 and a server 3. It is equipped with 00. The HMD200 and the server 300 communicate with each other via the network 101. To send and receive data.

[0019] In this embodiment, the server 300 is installed by a vendor that provides application-related services. This is a server device that receives request data from HMD200. It analyzes the data and provides the relevant services to the requesting HMD200. For example, Server 300, It holds various applications and, in response to requests from the HMD200, provides the HMD200 with It provides applications. Server 300 is, for example, an app store.

[0020] In this diagram, the HMD200 and server 300 connected to network 101 are Although only one of each is listed, the number of each is not specified. Also, if other devices are further... They may be connected and exchange data with each other.

[0021] First, application control by the application control system 100 of this embodiment The overview will be explained using Figures 1(b) to 1(e).

[0022] As shown in Figure 1(b), User 201 wearing the HMD200 writes on sign 501, etc. If you find a shape that appears to be an icon, turn on the HMD200's search mode. The exploration mode will be described later. At this time, user 201 will use HMD200 Ensure that the shape is included in the shooting area displayed on the screen. For example, shape Adjust the distance from the figure, your own orientation relative to the figure, etc.

[0023] Upon turning on the search mode, the HMD200 displays the image within user 201's field of view. An image is acquired. An example of the acquired image 601 is shown in Figure 1(c). Here, as an example... Then, point to the taxi stand.

[0024] The HMD200 then analyzes the acquired image 601 and identifies shapes and other elements within the field of view. Extract the region (hereinafter referred to as object) 602 on image 601. Note that Figure 1(d) The dotted rectangular area represents the extracted object 602. Details of the extraction process will be described later. .

[0025] When HMD200 extracts object 602, it identifies the object 602. It is determined that the selected icon has been identified, and as shown in Figure 1(e), the identified icon is... Launch the application that is associated with and holds the data. Here, it is the taxi dispatch app 603. This shows an example of how to launch the application. User 201 selects the desired process from the options and executes it.

[0026] Previously, the HMD200 displayed icons for all available applications that it had stored. It displays the options, selects the necessary icon from them, and then executes it. However, in reality... According to the implementation method, images of real objects within the user 201's field of view are used as icons. Identify and automatically launch that application. Therefore, user 201 has a large number of There's no need to search for and select the desired icon from among the icons; it's as if it already exists in the surrounding environment. It's as if actual shapes are being used as icons, and the relevant applications needed on the spot... You can activate the system.

[0027] The HMD200 of this embodiment, which implements the above processing, will be described below.

[0028] [Hardware configuration] Figure 2 is a hardware configuration diagram of the HMD200 of this embodiment. The 200 basically has the same configuration as a general-purpose computer (information processing device). The HMD200, as shown in this diagram, consists of a CPU 211, RAM 212, and ROM 21 3, camera 214, display 215, audio interface (I / F) 216 and , a communication interface (I / F) 217, a sensor 218, and a bus 219 connecting the parts. It is equipped with the following.

[0029] The camera 214, which is a shooting device, consists of an outward-facing camera 214a and an eye-tracking camera 214b. Prepare. The outward-facing camera 214a will include the field of view of the HMD200 user 201 in its shooting range. It can be installed. The gaze detection camera 214b is a camera that detects the direction of the user 201's gaze. Therefore, the gaze detection camera 214b captures the iris, pupil, etc. of the user 201's eye. It is mounted so as to be included in the shadow area.

[0030] Display 215 displays images acquired by camera 214 and images generated within HMD200. The display shows surface data. The display 215 is, for example, a transmissive liquid crystal device or an organic EL display. Devices, or MEMS (microelectromechanical systems) It consists of optical scanning devices using (ystems). However, the device This is not limited to this, but while displaying an image on the display 215, the direction of the display 215 Any device that can achieve a structure where the other side is visible would suffice.

[0031] The audio interface 216 includes, for example, a microphone, speaker, buzzer, etc. It receives external sound input and This includes sounds created within the HMD200, as well as voice and music sounds sent via the communication I / F217. This configuration performs output and output. In this embodiment, the audio I / F 216 may not be provided.

[0032] The communication I / F217 includes a coding circuit, a decoding circuit, an antenna, etc., and connects to network 101. Data is transmitted and received (data communication) with other devices via this interface. In this embodiment, the diagram is omitted. via access points, etc., or base stations, etc. of a mobile telephone network (not shown in the illustration) This is an interface that connects to network 101 via the communication interface 217. The MD200 performs data transmission and reception with each server 300 connected to network 101. .

[0033] The connection between the HMD200 and the access point is, for example, via Wi-Fi (registered trademark), etc. This is done by wireless communication or other communication methods. HMD200 and the mobile telephone communication Connection to the base station of the network is, for example, W-CDMA (registered trademark) (Wideband Cod e Division Multiple Access) method and GSM (registered trademark) Global System for Mobile communications) The method, LTE (Long Term Evolution) method, or other communication method It is done by formula.

[0034] Sensor 218 detects the current position, tilt, speed, and user 201 operations of the HMD200. Detect. Sensor 218 is, for example, a location information acquisition sensor such as a GPS receiver 218a. (Location information acquisition device), gyro sensor 218b, accelerometer 218c, geomagnetic sensor It is equipped with 218d, touch sensor 218e, etc. Note that it is not necessary to have all of the sensors 218. That's good too.

[0035] Note that RAM212 and ROM213 are combined unless there is a specific need to distinguish between them. This is collectively referred to as the memory device 230. Furthermore, it is not necessary to include an audio interface 216.

[0036] [Function Block] Figure 3(a) shows the HMD200 of this embodiment, related to the application control processing described above. Functional block diagram of the application control unit (also called the app control unit) 220, which is a function of the application control unit. The application control unit 220 includes an instruction reception display unit 221, an image processing unit 222, and an image resolution unit. It comprises an analysis unit 223, an application launch unit 224, and an application acquisition unit 225.

[0037] Each of these components is such that the CPU 211 processes the program stored in the memory device 230 into RAM. This is achieved by loading the program into memory (such as 212) and executing it.

[0038] Furthermore, the storage device 230 contains an icon database (icon DB) 231 and an application Application database (app DB) 232, image 233, display data 234, This is stored.

[0039] App DB232 is an icon, and when that icon is selected, it is launched. This is a database that registers applications in association with each other. Figure 4(a) shows App DB2. Here is an example of 32.

[0040] As shown in this figure, the application DB232 is associated with each icon. Application information is registered. In this embodiment, the icon is a graphic or symbol (and Below is the icon ID, which is information that uniquely identifies an icon shape (or simply an icon). 232a for each, the name of the application (app name) 232b, the application The app ID 232c is the information used to identify the application, and the location where the application is stored is also included. The information, application storage location 232d, is registered. Application storage location 232d is HMD2 This is the address in the memory of 00. Note that the application is the memory of the HMD200 If it is not stored locally, for example, on the network, the application will be stored. It could also be a URL that is the location of server 300.

[0041] Icon DB231 is a database for registering icons. Figure 4(b) Part 1 Here is an example.

[0042] As shown in this figure, the icons are, respectively, Icon ID 231a and Icon Shape 2 It comprises 31b, type (category) 231c, and content 231d. Icon ID 231 a is information that uniquely identifies the icon. Icon figure 231b is an icon. This is shape information for the figure to be displayed on display 215.

[0043] Type 231c is pre-configured for each processing content of the application associated with the icon. A defined classification is registered. In this embodiment, for example, transportation, payment, facilities, goods, shopping. This includes games, music and video streaming, etc.

[0044] Content 231d is a description of icon figure 231b and / or is associated with the icon. The application's processing details are registered. The information registered in content 231d is: This is used to analyze the meaning of the captured images and identify related icons, as described later.

[0045] Furthermore, the storage device 230 stores the image 233 acquired by the imaging processing unit 222. Furthermore, the instruction reception display unit 221 generates screen data to be displayed on the display 215. Various types of data used during this process are stored as display data 234.

[0046] The instruction reception display unit 221 receives operation instructions from the user 201. Furthermore, the processing of each part... Screen data is generated from the calculation results and displayed on display 215.

[0047] The instruction reception display unit 221 displays, for example, various operation instruction buttons (areas) on the display 215. ) is displayed and the selection by user 201 is accepted. The data for the operation instruction buttons is pre-written. It is stored in the memory device.

[0048] Instructions such as selections by user 201 are given, for example, by eye gaze. It detects 201 gaze directions and uses them as a substitute for a pointing device. Display It is determined that the area located in the line of sight above 215 has been selected.

[0049] Existing technology will be used to detect the direction of gaze. For example, at a stationary reference point of user 201's eye In contrast, detection is performed based on the position of the moving point. A fixed reference point can be used, for example, the inner corner of the eye. The moving points include, for example, the pupil and iris. These positions are determined by the gaze detection camera 2. This is obtained by analyzing the images captured by 14b. For example, to detect the selection instruction, A predetermined number of blinks may be added.

[0050] Here, the function block of the instruction reception display unit 221, which primarily realizes instructions by eye gaze, The control is shown in Figure 3(b). As shown in this figure, the instruction reception display unit 221 is connected to the display control unit 24 The system comprises 1, a display image generation unit 242, a gaze detection unit 243, and an analysis control unit 244.

[0051] The display image generation unit 242 generates screen data to be displayed on the display 215. The data is generated, for example, using various data stored in the storage device 230.

[0052] The display control unit 241 receives the screen data generated by the display image generation unit 242 and displays it on the display 21. Display it in 5.

[0053] The gaze detection unit 243 detects the direction of gaze from the video captured by the gaze detection camera 214b. .

[0054] The analysis control unit 244 analyzes the gaze direction detected by the gaze detection unit 243 and identifies the instruction. Signals are output to each part. The analysis control unit 244, when the line of sight direction is the same for a predetermined period of time, It is determined that the image region located in the line of sight has been selected. Also, if the line of sight direction is for a predetermined period of time, If the images are moving in the same direction and blink a predetermined number of times at predetermined intervals, the image region is selected. It can be identified as an object.

[0055] Note that the operation instructions include, for example, an instruction to set the HMD200's processing to search mode (search instruction). ), select the desired icon or application from the extracted group of icons. Instructions include execution instructions that command the execution of a selected application, etc. Instruction reception display. When unit 221 receives each instruction, it sends a selection instruction signal and an execution instruction signal respectively. Output.

[0056] Note that the search mode is a mode that captures the actual field of view of user 201. Instruction reception display unit When 221 receives a search instruction, it instructs the image acquisition processing unit 222 to acquire an image. Outputs a shadow indication signal. Outputs a shooting instruction signal until the search mode termination instruction is received. .

[0057] For example, the search instruction is given by the user 20 to press the search instruction button displayed on the display 215. The selection is accepted when 1 is selected. The search command button is, for example, when the HMD200 is started. Subsequently, or after receiving the instruction to start the application control process, display 2 It will be displayed on page 15.

[0058] The selection instruction is, for example, from among several icons displayed on the display 215, the desired This is an instruction to accept the selection of an icon. An execution instruction is an instruction to run the application. The application associated with the icon selected by the selection instruction will be executed. We accept instructions.

[0059] When the imaging processing unit 222 receives an imaging instruction signal from the instruction reception display unit 221, it performs the following at a predetermined time At intervals, the field of view (actual field of view) of user 201 is captured. The capture is performed by the outward-facing camera 214a. The image 233 acquired by the camera is stored in the storage device 230.

[0060] Note that the imaging processing unit 222 takes an image once when it enters search mode. It may be configured in this way.

[0061] The image analysis unit 223 stores the image 233 acquired by the image capture processing unit 222 in the storage device 230. Then, the image 233 is analyzed, and the icons within the image 233 are extracted.

[0062] Specifically, the image analysis unit 223 first analyzes the actual objects present in the external environment within the image 233. The region corresponding to the object (real object) is extracted as an object in the image. Then, using region information, which is information that identifies the region of the extracted object, the actual object Identify the icons associated with the object.

[0063] Object extraction is performed, for example, by known contour detection (edge ​​detection). For all extracted objects, icon shape 2 is stored in icon DB231. Determine whether it matches 31b.

[0064] Discrimination is performed, for example, by a known matching method based solely on shape. Then, the extracted objects Within the project, the object that matches the icon shape 231b held in icon DB231 The object is identified as an icon. Then, the matching icon shape 231b is assigned to the object. Identify it as an icon related to the project.

[0065] Note that the term "match" here is not limited to an exact match, but includes matching results that exceed a predetermined similarity level. nothing.

[0066] Furthermore, the objects that the image analysis unit 223 extracts from image 233 as objects are those on the street. The displayed mark, the mark posted or displayed in the store, or the mark printed on the product tag It can be any real object, such as a mark. Also, it can be something that is painted, Printed on a ball-shaped object, or on a display device such as a tablet or information board. This includes various forms, such as those shown.

[0067] Furthermore, if there are multiple objects that have been determined to be icons, the image analysis unit 223 will determine that... These are displayed on the display 215. Then, via the instruction reception display unit 221, the selected instruction The display accepts the information. At this time, the display 215 will show the object. It may be the matching icon shape 231b, or the image data of the extracted object's region. It's fine to use the data itself.

[0068] The icon may also be displayed directly on the retina of the eye. Shooting range of the outward-facing camera 214a Any display method is acceptable as long as it can display the external scenery present in a way that is visible to the naked eye.

[0069] When the instruction to select an icon is received, the instruction reception display unit 221 will select Selected icons are displayed in a different manner than unselected icons. For example, their size can be changed. This includes changing the color, surrounding it with a special frame, etc. Additionally, it allows you to enlarge only the selected icon. Display and make the size of other icons smaller compared to the selected icon. For example, display only the selected icons in color, and display other icons in grayscale. That's fine.

[0070] The application launch unit 224 executes the application associated with the selected icon. Specifically, the application launch unit 224 refers to the application DB 232 and selects an eye. Identify the application associated with the selected icon. Extract the application storage location 232d that is registered in association with icon ID 231a / 232a. It then loads the application's program into RAM212 from that location and starts it up. To make someone do it.

[0071] Furthermore, the application launch unit 224 will load the selected icon's application into the storage device 23. If it is not stored in 0, it sends an application acquisition signal to the application acquisition unit 225. For example, the selected icon's app storage location 232d is on the UR on network 101. If L is registered, or if the selected icon is in app DB232, This is the case when ID232a is not registered. The app acquisition signal will include icon ID232a. Include. If a URL is registered in the app storage location 232d, also include the URL. ru.

[0072] When the app acquisition unit 225 receives an app acquisition signal from the app launch unit 224, the app The application associated with icon ID 232a included in the acquired signal is on server 3 Get it from 00. If a URL is received, access that URL and the application Get the information.

[0073] At this time, the application acquisition unit 225 acquires the application including icon ID 232a. The request is sent to the server 300 and the application is provided. Download the application.

[0074] Furthermore, the application acquisition unit 225 stores the provided application in the storage device 230. Then, the storage location is associated with the icon ID 232a and stored in the application storage area of ​​application DB 232. The device is registered in location 232d. Then, the application launch unit 224 is notified that the registration is complete. The application launch unit 224 receives the notification and launches the application.

[0075] By registering the downloaded application in the application DB232, the next time... You can then continue using the application as is.

[0076] Furthermore, if one object is determined to be an icon by the analysis performed by the image analysis unit 223, In some cases, the above selection instruction may not be given. In this case, the image analysis unit 223 will extract When the displayed object is determined to be an icon, the application launch unit 224 will... The application associated with the icon will be executed directly.

[0077] Furthermore, although not shown in the figures, the HMD200 of this embodiment also includes the operation of each hardware component. It includes a control unit that controls the operation of the communication I / F 217. Specifically, it includes a communication control unit that controls the operation of the communication I / F 217. This includes an outward-facing camera control unit that controls the operation of the outward-facing camera 214a, and a unit provided for each sensor 218. This includes a sensor control unit that controls the operation of each of the sensors 218.

[0078] The communication control unit utilizes the WiFi, Bluetooth®, and 4 other technologies available in the HMD200. Controlling wireless communication functions of mobile communication systems such as G and 5G, and sending and receiving data with external devices. To realize trust. The sensor control unit outputs the signal detected by each sensor.

[0079] [Processing flow] Next, the flow of application control processing by the application control unit 220 of this embodiment will be explained. Figure 5 shows the processing flow of the application control process in this embodiment. The process begins upon receiving a search instruction.

[0080] The image processing unit 222 causes the outward-facing camera 214a to capture an image that includes the user's field of view 201. Image 233 is acquired (step S1101). The acquired image 233 is stored in the storage device 230. Remember.

[0081] The image analysis unit 223 analyzes the image 233 and extracts objects (step S 1102). Then, determine whether the extracted object is an icon or not, and the icon If so, identify the icon (step S1103). Here, icon DB23 Refer to 1, and an icon figure 231b having a similar shape to the object is registered. The decision is made based on whether or not it exists. In other words, the extracted object has a similar icon shape 23. If 1b is registered in icon DB231, the object will be treated as a similar icon. It is identified as a con. And icon shape 231b is identified as a similar icon. By extracting icon ID 231a which is registered in association with the icon, the icon can be specially selected. Determine. Furthermore, during the analysis of image 233, the image analysis unit 223 determines the object where the icon was identified. The number of objects is counted as the number of icons.

[0082] The instruction reception display unit 221 determines whether the number of icons is 0 or not (step S110 4) If the number of icons is 0, terminate the process.

[0083] On the other hand, if it is not 0, the instruction reception display unit 221 determines whether the number of icons is 1 or not. (Step S1105).

[0084] If the number of icons is 1 (S1105; Yes), proceed to step S1108 described below. do.

[0085] On the other hand, if the number of icons is not 1 (S1105; No), that is, if the number of icons is multiple If so, the instruction reception display unit 221 will show all the identified icons and the corresponding icons The icon shape 231b is displayed on the display 215 (step S1106). , accept selection instructions (step S1107).

[0086] The application launch unit 224 refers to the application DB 232 and selects the icon I Applications associated with D231a / 232a are registered within the HMD200. Determine whether it is present or not (step S1108). Here, the application storage location is 232d. If an address is stored in the storage device 230, the application is registered. I conclude that they are there.

[0087] If an application is registered in the HMD200, then apply that application The process starts (step S1109) and then terminates.

[0088] On the other hand, if no application is registered in the HMD200, the application acquisition unit 2 The application acquisition unit 225 is instructed to acquire the application. The application acquisition unit 225 accesses the server 300. Search by clicking (step S1111), and associate the icon with icon ID 232a. The application is retrieved (downloaded) from server 300 (step S11). 12). Then, the app acquisition unit 225 receives the downloaded application via Aiko. Register it in application DB232, associating it with ID232a (Step S11 BR>P3) Then, proceed to step S1109.

[0089] In this embodiment, during the search mode, the application control unit 220 performs the following actions at predetermined time intervals: Execute the above process.

[0090] Furthermore, when transitioning between each process, the previous display will be cleared and the new display will be shown. This prevents unsightly displays caused by unnecessary information. Of course, this method of screen display is... This is applicable to all screen transitions in this embodiment.

[0091] [Usage example] Next, Figures 6(a) to 6(a) show specific examples of how the application control system of this embodiment can be used. Let's explain using 7(c). Here, user 201 is near a taxi stand, Let's take the example of launching a ride-hailing application. This is an application for requesting, booking, and searching for taxi fares. That is the case.

[0092] As shown in Figure 1(b), when user 201 arrives near a taxi stand, I see sign 501 with the same shape as the icon for the ride-hailing application. When user 201 sees sign 501, their gaze is directed to display on display 215 Press the search command button 711.

[0093] Figure 6(a) shows what user 201 sees through display 215 in this case. As mentioned above, this display 215 is transparent. Therefore, to user 201 This shows the image generated by the HMD200 superimposed on the real image 510 in real space. In this case, the real image of the real space (real environment) is represented by solid lines, and the image generated by the instruction reception display unit 221 is broken. It is shown by a line. In this diagram, the search instruction button 711 is an image.

[0094] This is, for example, a station square. User 201's field of view includes real objects in real space. As an example, a bus stop with the bus symbol 511 and a train platform with the train symbol 512 are proposed. Inside, a sign with a taxi shape 513, a person 514, a flower 515, a weed 516, a dog 51 There is a 7th place.

[0095] When it is confirmed that user 201 has selected the search instruction button 711 by eye gaze, the instruction The reception display unit 221 instructs the image processing unit 222 to acquire an image. The field of view is pre-configured to include the user's actual field of view via the HMD200's display 215. The image 610 acquired at this time is shown in Figure 6(b).

[0096] The image analysis unit 223 extracts objects from this image 610. An example of an object is shown in Figure 6(c). From image 610, which was taken in such an environment, a train Figure 611, Bus Figure 612, Taxi Figure 613, Person Figure 614, Flower Figure 615, a weed shape 616, a dog shape 617, a circle 618, an ellipse 619, etc. are objects. The image analysis unit 223 then analyzes each of the extracted objects. Then, refer to icon DB231 to determine whether or not it is an icon.

[0097] In other words, the image analysis unit 223 processes these objects, respectively, into the icon DB2. Compare it with icon shape 231b registered in 31 and check if there are any matches or similarities. Determine whether it is true or false. Then, the icon shape 231b registered in icon DB231 If similar objects are found, the extracted object will be identified as icon shape 231b. do.

[0098] In the example above, the bus figure 611, the train figure 612, and the taxi figure 613 are A The icons are identified, and each icon ID 231a is specified.

[0099] Then, the instruction reception display unit 221 displays the icon shape 231b of each identified icon, Extract from icon DB231 and generate as display icon 712. Figure 7(a) shows an example of what CON 712 displays on the display 215. At this time, the display icon The display area of ​​n712 is predetermined. The display area will be determined according to instructions from user 201. It may be made more modifiable.

[0100] Then, when a selection instruction is received from user 201, it is associated with the corresponding icon. Launch the application. Here, the taxi icon is selected. The corresponding taxi dispatch application is then launched.

[0101] At this time, the instruction reception display unit 221 displays the display icon 712 that has received the selection instruction, and The display icon 712 is displayed in a different manner. For example, as shown in Figure 7(b), Display this icon larger than the other 712 display icons.

[0102] Subsequently, the instruction reception display unit 221, in accordance with the instructions from the application launch unit 224, as shown in Figure 7(c As shown in the image, the initial screen data 713 of the launched application is displayed on the display. Display it at 215.

[0103] As described above, the HMD200 of this embodiment is directed outwards according to the instructions of the user 201. The image 233 captured by camera 214a is analyzed to extract real objects present in the external environment. Then, identify the corresponding icon for each. And then associate it with the identified icon. The application is launched. In other words, in this embodiment, a shape similar to an icon is used. The application launches simply by taking a picture of a real object.

[0104] Thus, according to this embodiment, various objects placed in the external environment are treated as if they were A It is used like an icon to launch the corresponding application. Therefore, in this embodiment Therefore, in the environment where user 201 is located, the application that is most likely to be used will be... It can be launched without bothering user 201. From the display of numerous icons... Since there is no process to accept selections, it does not obstruct the view of user 201, and convenience is maintained. We can provide the HMD200, which offers excellent operability.

[0105] Furthermore, according to this embodiment, in the environment in which user 201 is located, it is highly likely that it will be used. Even if multiple application icons are identified, only those will be displayed. In other words, only icons that are likely to be used are selectively displayed.

[0106] Generally, in devices built on smartphones, tablets, and personal computers, All the icons for the applications available on that device are displayed, and from there... User 201 needs to make a selection. Therefore, many icons appear on display 215. It will be displayed.

[0107] However, according to this embodiment, only the minimum necessary icons are displayed. Therefore, even with a display terminal with a small display area, such as the HMD200, the field of view of user 201 It does not interfere with the process. Also, it reduces the effort of searching for the desired icon among many icons. It can also be simplified, making it easy to use. Therefore, it is an HMD with high visibility, convenience, and operability. I can offer 200.

[0108] Furthermore, in this embodiment, the application that user 201 wants to run is displayed on the HMD200. If not registered, the application acquisition unit 225 acquires it from the server 300 and registers it. Therefore, the probability of the desired application running is improved.

[0109] <Example 1> Furthermore, in the above embodiment, the instruction reception display unit 221 receives instructions from the user 201, It accepts instructions by analyzing the gaze of The 201. However, the method for receiving instructions is However, it is not limited to this. For example, sensing using the touch sensor 218e may be used. Within the display area of ​​the display 215, cursor movement is performed by operations on the cursor display and The area is selected by a selection action, such as a double tap on the touch sensor 218e. I acknowledge and accept that this has been done.

[0110] Furthermore, as shown in Figure 8(a), the instruction reception display unit 221 displays the finger movements of the user 201. This is detected by the outward-facing camera 214a. If a gesture operation is detected as a finger movement, The instruction reception display unit 221 may determine that the display icon 712 has been selected. Note that gesture operation refers to, for example, tapping the area where display icon 712 is displayed. These include actions like poking, clicking, or pinching. Touching actions occur when the fingers and the displayed icon 712 overlap in the line of sight, as well as when the object Touching the actual device is also acceptable.

[0111] <Modification 2> In the above embodiment, as shown in Figure 7(a), the display icon 712 is displayed Ray 215 is displayed in a predetermined area, but is not limited to this. For example, the instruction receiver As shown in Figure 8(b), the attached display unit 221 displays the user 201's information on the display 215. Display icon 712 at the location of the original real object in the real field of view of the virtual object It may be displayed in this way.

[0112] At this time, the icon of the corresponding application stored in the storage device 230, The display method of icon 712 may be changed to distinguish it from other icons.

[0113] Furthermore, it is not necessary to display icon 712 as icon shape 231b. Good. In the area of ​​the object identified as an icon on display 215, An indication that the object has been identified as an icon may be provided. For example, a dotted line. This includes displaying a frame around the object at its location.

[0114] <Variation 3> Furthermore, the display of this icon 712 can be shown or hidden at the instruction of user 201. It may be configured to be switchable. If display icon 712 is not displayed, user 201 This involves selecting the area corresponding to the actual object as an icon, and then using the corresponding application. It may also be configured to start up. In this case, the instruction reception display unit 221 captures the field of view. By analyzing the image, the object in the image region selected by user 201 can be identified. It identifies icons and accepts instructions. It also determines if the display is interfering with the perception of the outside world. If this occurs, the instruction reception display unit 221 may prohibit the display of the display icon 712. In this case, the instruction reception display unit 221 acquires an image of the area corresponding to the actual field of view before displaying, and The image analysis unit 223 performs the analysis, and the display icon 712 indicates the group of real objects in the outside world. It determines whether or not the recognition of is impaired. Also, according to the analysis results of the image analysis unit 223, The display unit 221 is designed so as not to hinder the recognition of real objects in the outside world as much as possible. You may also adjust the settings to display icon 712 in the appropriate area.

[0115] <Modification 4> Furthermore, in the above embodiment, the image analysis unit 223 analyzes the acquired image 233 to determine the user 2 The shapes within the real field of view of 01 are extracted as objects. However, the objects to be extracted The term "object" is not limited to shapes. For example, it could be a string of characters displayed on a sign, etc. Examples of the strings 621, 622, and 623 to be extracted are shown in Figures 9(a) to 9(c).

[0116] String extraction is performed using existing technologies, such as OCR. Then, image analysis is performed. Section 223 analyzes the meaning of the extracted string. Semantic analysis is performed using existing techniques, such as morphological analysis. This is done through processes such as raw analysis. Then, the meaning of the analysis and the contents of icon DB231 231 It compares with d and determines whether the corresponding icon is present or not. In other words, the image analysis unit 223 solves If the meaning obtained by the analysis matches the content 231d with a predetermined accuracy, then the content 231 Icons containing "d" are identified as related icons.

[0117] Furthermore, if the image analysis unit 223 identifies an icon related to the extracted string, The icon shape 231b of the icon is extracted and displayed icon 7 on the instruction reception display unit 221. 12 may be displayed on display 215. The display position is within the actual field of view of the string. This is the vicinity. Examples of display on display 215 are shown in Figures 9(d) to 9(f). Here Figure 9(f) shows an example where multiple icons are identified as related icons. In this case, all identified icons (icon shape 231b) will be displayed.

[0118] By not limiting the extracted objects to shapes, the user 201 can see a wider range of objects. The area corresponding to the icon can be extracted from the field. Accordingly, the desired application The probability of the system being activated increases. Therefore, convenience for user 201 improves. .

[0119] <Modification 5> In the above embodiment, if there is one icon, the corresponding application will be automatically It is being started. However, it is not limited to this. For example, before starting, user 201 can actually It may be configured to accept line instructions (execution instructions). That is, step S11 above Before step 09, the application launch unit 224 receives an execution command and launches the application. .

[0120] At this time, as shown in Figure 8(c), the instruction reception display unit 221 is, for example, a display An instruction button 715 that accepts execution instructions is displayed in a designated area of ​​215. The system accepts the selection of the instruction button 715 by gaze as an execution instruction.

[0121] By accepting execution instructions from user 201 before startup, unnecessary applications are filtered out. This can reduce startup time.

[0122] If you choose to enable automatic startup, after automatic startup, user 201 will initiate the application launch. Alternatively, the system may be configured to accept instructions to stop the operation midway (cancel instructions). The cancellation instruction is, for example, a gesture operation by user 201, as explained using Figure 8(a). You can use gestures, etc. Gesture operation can be an operation such as waving your hand or shaking your head. good.

[0123] Furthermore, this gesture may be configured to cancel unintended displays. For example, user 201 may unintentionally select the search instruction button 711 due to their gaze. There is a case where the display icons 712 and initial screen data 713 that are displayed afterwards are This is unintended. In such cases, the instruction reception display unit 221 receives the cancellation instruction. Pressing this button will cancel the application control process and return to the original screen.

[0124] Furthermore, when launching automatically without accepting execution commands, depending on the application... Therefore, you may set it to be disabled in advance. Specifically, the relevant application DB232 A flag indicating that the application should not be launched is set in the application's record. (Application launch section) 224 If startup is disabled, in step S1109 above, the application The system will not launch, and instead, for example, it will notify user 201 that launching the system is prohibited. .

[0125] <Variation 6> In the above embodiment, in the application DB232, one icon ID232a In contrast, one application is associated with it, but it is not limited to that. For example, Even if multiple applications are associated with a single icon ID 232a, stomach.

[0126] For example, the icon for the shape of a taxi (icon ID 232a is Taxi 01) App D when several taxi company ride-hailing applications are associated with each other. An example of B232 is shown in Figure 10(a).

[0127] In this case, the application launch unit 224 receives the icon ID 23 of the icon that received the selection instruction. Information on all applications associated with 2a is output to the instruction reception display unit 221. For example, it outputs the application name 232b. Upon receiving this, the instruction reception display unit 221 displays Figure 1 As shown in 0(b), image data that can identify the associated application (app Generate (Re-icon) 714 and display it on display 215.

[0128] The instruction reception display unit 221 receives instructions from the user 201 regarding the selection of the application icon 714. The application launch unit 224 then accepts the request and launches the application.

[0129] Note that multiple applications can be associated with a single icon ID 232a. If present, set a priority for the application and assign the app icon 714 to priority. It may also be configured to display them in descending order of priority. In this case, the priority is set in the application DB23 beforehand. Register it as option 2.

[0130] If a priority order is set, the instruction reception display unit 221 will show instructions from user 201. Without waiting for the selection, it may be determined that the option with the highest priority has been selected.

[0131] Also, if the compatible application is not registered in the app DB232, app retrieval will fail. Section 225 obtains the data from server 300 as described above. At this time, on server 300, Sometimes, multiple applications are available for a single icon. In this case, Before downloading, the instruction reception display unit 221 displays information that identifies each application. It is configured to display on display 215 and accept selection instructions from user 201. That's good too.

[0132] Furthermore, when the application acquisition unit 225 acquires an application from the server 300, In some cases, there may be no application associated with the trusted icon ID232a. In such cases, the AI ​​(Artificial Intelligent) on server 300 will be used. The ence function allows you to sort highly relevant (suitable) applications by relevance (suitability). A predetermined number of selections may be made in descending order of their scores.

[0133] The 714 app icons for the selected applications are displayed in order of suitability. This is also acceptable. Furthermore, the display may be sorted by popularity. And, the selection from user 201... Upon receiving the notification, the application acquisition unit 225 will send the selected application to the server 30 Download from scratch.

[0134] <Example 7> Furthermore, it works in conjunction with other functions of the HMD200, and also displays icon 712 or A The number of pre-icons displayed (714) may be reduced. Also, the number of icons to be matched may be reduced. The processing load on the image analysis unit 223 may be reduced. This will increase the processing speed. Along with increasing the probability of being identified as an icon and the probability of identifying the desired icon, It is possible.

[0135] For example, depending on the status of user 201, the icons to be matched in icon DB231 This restricts the process. In this case, the image analysis unit 223 first analyzes the image using the above method to find objects. Extract. Then, when determining whether or not it is an icon, select type 231c that matches the situation. Alternatively, it will only match with the icon shape 231b of the icon that has content 231d.

[0136] For example, if the HMD200 has a schedule management function, even if you use that information... Good. For example, if the schedule management function makes it clear that you have a train to take, The image analysis unit 223 refers to content 231d or type 231c and selects train-related icons. Only the icon graphic 231b is extracted and collated. Icons related to trains, for example, are icons registered in the content 231d as being related to trains, or icons whose type 231c is transportation. Note that the image analysis unit 223 obtains schedule information from the schedule management function and specifies the collation range through semantic analysis. content 231d registered as being related to trains, or icons whose type 231c is transportation. Note that the image analysis unit 223 obtains schedule information from the schedule management function and specifies the collation range through semantic analysis. content 231d registered as being related to trains, or icons whose type 231c is

[0137] Also, for example, when the HMD 200 has a function of accumulating processing history, this information may be used. Also, when the HMD 200 has a function of accumulating the processing history in association with the position information at the time of processing, this information may be used. These may be further used in combination with the schedule information. Also, for example, when the HMD 200 has a function of accumulating processing history, this information may be used. Also, when the HMD 200 has a function of accumulating the processing history in association with the position information at the time of processing, this information may be used. These may be further used in combination with the schedule information. Also, for example, when the HMD 200 has a function of accumulating processing history, this information may be used. Also, when the HMD 200 has a function of accumulating the processing history in association with the position information at the time of processing, this information may be used. These may be further used in combination with the schedule information. Also, for example, when the HMD 200 has a function of accumulating processing history, this information may be used. Also, when the HMD 200 has a function of accumulating the processing history in association with the position information at the time of processing, this information may be used. These may be further used in combination with the schedule information.

[0138] For example, if it is recorded in the history that the user 201 pays with electronic money at a specific store, the image analysis unit 223 uses this information to limit the collation range. The information for specifying the store uses the position information of the store. The position information of the store is obtained by the GPS receiver 218a which is a position information acquisition device. For example, if it is recorded in the history that the user 201 pays with electronic money at a specific store, the image analysis unit 223 uses this information to limit the collation range. The information for specifying the store uses the position information of the store. The position information of the store is obtained by the GPS receiver 218a which is a position information acquisition device. For example, if it is recorded in the history that the user 201 pays with electronic money at a specific store, the image analysis unit 223 uses this information to limit the collation range. The information for specifying the store uses the position information of the store. The position information of the store is obtained by the GPS receiver 218a which is a position information acquisition device. For example, if it is recorded in the history that the user 201 pays with electronic money at a specific store, the image analysis unit 223 uses this information to limit the collation range. The information for specifying the store uses the position information of the store. The position information of the store is obtained by the GPS receiver 218a which is a position information acquisition device.

[0139] That is, when the current position of the HMD 200 is the said store, the image analysis unit 223 collates the extracted object only with the icon graphic 231b related to electronic money in the icon DB 231. For example, icons in which information indicating that it is electronic money is registered in the content 231d, or icons in which settlement is registered in the type 231c, etc. That is, when the current position of the HMD 200 is the said store, the image analysis unit 223 collates the extracted object only with the icon graphic 231b related to electronic money in the icon DB 231. For example, icons in which information indicating that it is electronic money is registered in the content 231d, or icons in which settlement is registered in the type 231c, etc. That is, when the current position of the HMD 200 is the said store, the image analysis unit 223 collates the extracted object only with the icon graphic 231b related to electronic money in the icon DB 231. For example, icons in which information indicating that it is electronic money is registered in the content 231d, or icons in which settlement is registered in the type 231c, etc. That is, when the current position of the HMD 200 is the said store, the image analysis unit 223 collates the extracted object only with the icon graphic 231b related to electronic money in the icon DB 231. For example, icons in which information indicating that it is electronic money is registered in the content 231d, or icons in which settlement is registered in the type 231c, etc. That is, when the current position of the HMD 200 is the said store, the image analysis unit 223 collates the extracted object only with the icon graphic 231b related to electronic money in the icon DB urchased.

[0140] Note that furthermore, it is recorded that at a specific store, payment is made with a specific electronic money. In this case, the app launch unit 224 will activate when user 201 turns on the search mode within the store. In addition, configure it to directly launch the application related to that specific electronic money. The application launch unit 224 receives location information and search instructions via the instruction reception display unit 221. Receive.

[0141] Furthermore, if information about the nature of the current location can be obtained in addition to the current location information. Depending on the information, the range of icons to be extracted may be limited.

[0142] For example, if your current location is a government-related facility, the icon will show administrative services, and transportation will be displayed. If the target is Seki, the search range of the image analysis unit 223 will be limited to only transportation-related icons.

[0143] This means that even if you inadvertently photograph a real object that resembles an icon, the icon will still be recognized. It is not extracted as such. Note that information about the current location is obtained from the GPS receiver. The current location information obtained by 218a, the mobile communication function of communication I / F 217, etc. The decision is made using current location information and map information, etc.

[0144] Furthermore, it may be linked with other databases. For example, as shown in Figure 11, each store can be pre-programmed. The applications available to be used are stored in a database (store DB235), and This may be used to reduce the number of icon shapes 231b to be matched.

[0145] Store DB235 contains information for each store, including its location and the applications available to it. ID232c or icon ID232a is registered. In the example in Figure 11, icon I The following is an example of when D232a is registered. The image analysis unit 223 receives the GPS signal of the HMD200. If the current location information obtained by the 218a device matches the store's location information, then the store will be contacted. It is determined that it is present. Then, the available icon ID 232a is registered in store DB235. This limits the matching range.

[0146] Furthermore, if store DB235 exists, and it is determined that user 201 is in a store, When the search mode is turned ON, the instruction reception display unit 221 sends the image analysis unit 223 to analyze the image. Instead of analyzing it, icon shape 2 of icon ID 232a is registered in store DB235. Alternatively, you can extract 31b from icon DB231 and display all of them on display 215. .

[0147] Additionally, the store database 235 includes a priority list for available applications for each store. This may be set. In this case, the instruction reception display unit 221 will be in search mode within the store. When this is turned ON, the application registered on the HMD200 will be given the highest priority. A high-level application may be launched on the application launch unit 224.

[0148] In particular, applying such modifications to application control processing during payment, for example, The HMD200 displays icons corresponding to each real object present in the field of view. Even if you have a relationship, icons unrelated to payments will not be displayed. This results in only the most relevant options being selected, making it even more beneficial for the user. It improves usability.

[0149] Note that in this modified example, multiple icons of the same type are displayed, and the user selects one according to the prescribed rules. Applicable to all. The prescribed rules include, for example, past usage frequency and predetermined priority order. It is as follows.

[0150] <Modified Example 8> Also, when an auxiliary code such as a QR code (registered trademark) is arranged near an object corresponding to an icon, in order to reduce the processing load of the application control unit 220, the auxiliary information provided by the auxiliary code may be used. For example, the application may be acquired according to the information of the auxiliary code. Hereinafter, the case where the auxiliary code is a QR code will be described as an example. When an auxiliary code such as a QR code (registered trademark) is arranged near an object corresponding to an icon, in order to reduce the processing load of the application control unit 220, the auxiliary information provided by the auxiliary code may be used. For example, the application may be acquired according to the information of the auxiliary code. Hereinafter, the case where the auxiliary code is a QR code will be described as an example. In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233. In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233. In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233.

[0151] In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233. In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233. In the present embodiment, when the object extracted by the image analysis unit 223 is determined to be an icon, the image analysis unit 223 further refers to the auxiliary information of the QR code near the object in the image 233.

[0152]

[0153]

[0154]

[0155] <00,01001>

[0156]

[0157] [[ID=,57]]

[0158]

[0159]

[0160]

[0161]

[0162] )

[0163] [[ID=9))

[0164]

[0165] <0001)62>

[0166]

[0167]

[0168]

[0169]

[0170] [[ID=)40]] <00010()5>

[0172]

[0173] )

[0174] [[ID=,64]]

[0175] <0001IZ5> [[ID=I76]]

[0177] <()001I37>

[0178] <()001139> <0001I42> <000114The QR code itself may be converted to text and displayed on the display 215.

[0155] For example, as shown in Figure 12(a), on a sign 501 with a taxi shape 513 attached It is assumed that a QR code 531 is attached. The image analysis unit 223 takes an image of this field of view. From image 233, identify the area corresponding to the taxi shape 513 as an icon. Then, the image Identify the nearest QR code from among 233, obtain its information, and use it as described above.

[0156] As shown in Figure 12(a), the QR code is placed on the taxi stand sign 501. This includes, for example, the URL information of a company providing taxi dispatch services, and taxi dispatch apps. URL information of the application, location information indicating the designated pick-up location, telephone number information of the pick-up center, These are recorded.

[0157] Additionally, QR codes placed near the icon for type 231c, which is shopping, will include For example, the URL information of an online shop application, store information, etc. URL information, store product information, payment information at checkout, etc., are recorded.

[0158] Additionally, the QR codes placed near the bus route diagrams contain bus route information, boarding location information, and more. URL information for websites that publish fare information, etc., URL for websites that provide real-time service information Information such as L-numbers and URLs of websites providing transit directions are recorded.

[0159] Additionally, the QR code placed near the image representing the toilet will show the location of the public toilet and the current The URL information of a website that provides directions from your current location to the nearest public restroom is recorded. Such QR codes are installed, for example, in large stores and shopping malls. .

[0160] In addition to the auxiliary information obtained via the QR code, the application launch unit 224, as in the above embodiment, However, refer to application DB232 and the application is registered in association with an icon. You may start it up.

[0161] Furthermore, auxiliary information may be used in processing the launched application.

[0162] In this case, the launched application, along with the auxiliary information it read, or the auxiliary information Based on the report, the application performs the prescribed processing. This allows the auxiliary information obtained from the QR code to be processed. It can be used as a parameter when executing commands, enabling the automation of more complex processes. Therefore, by using auxiliary code in addition to the processing operations performed by the application software, This makes it possible for HMD200 user 201 to obtain even more desirable processing.

[0163] For example, as shown in Figure 12(b), the tags attached to clothing etc. have the mark 518 and QR code. It shall be marked with code 531. This mark 518 is for online stores that handle clothing. Assume that the store application is associated. Also, QR code 531 This refers to information that identifies the garment, such as the manufacturer's code, product number, color, and size. It shall be assumed that it is registered as a report.

[0164] When the app is launched, it automatically retrieves supplementary information to find the same product as the clothing in question. You can order it from the online store. Also, you can search for products in different colors and sizes for the same clothing item. It can be searched.

[0165] Also, for example, as shown in Figure 13(a), near the store's cash register (POS) 542 Typically, this includes the logos of accepted credit cards and accepted electronic money. Other available payment methods will be indicated by symbols, etc. In such areas, receipt 5 You may also use the QR code 531 described in 41, etc. In this case, the QR code 531 will be The total purchase amount, store information, purchased item information, payment institution, and other information necessary for payment are listed as supplementary information. It is recorded as follows: The payment application launched by selecting the icon is The payment process can be carried out using the auxiliary information recorded in QR code 531. Alternatively, a QR code may be displayed on a tablet or other display device instead of receipt 541. Furthermore, it displays the logos of usable credit cards, usable electronic money, and so on. In addition, the terminal may display symbols or other indications of other available payment methods.

[0166] Note that the field of view and the receipt 541 may be photographed separately. In this case, The application control unit 220 extracts and identifies from the image of the user 201's field of view that was captured earlier. After launching the application corresponding to the icon, take a picture of receipt 541 and scan the QR code. Retrieve 531 auxiliary information and reflect it in the application's processing.

[0167] Note that the above explanation assumes that the image analysis unit 223 also functions as a QR code reader. This was explained using the example. However, the QR code reader function is separate from the image analysis unit 223. It may be provided. In this case, after the image analysis unit 223 identifies the QR code, the The function is instructed to read.

[0168] Furthermore, the above explanation states that QR code reading is performed automatically. However, The read operation may be configured to occur only when an instruction is received from user 201. In this case, the instruction reception display unit 221 receives an instruction from the user 201, and the image analysis unit Instruct 223 or the reader function to read. For example, if user 201's gaze direction is If the orientation of the QR code on the image displayed on display 215 matches, it is determined that it has been selected. do.

[0169] Note that in the above example, the object and the QR code are independent. However, There are QR codes with icons that combine a graphic shape with an auxiliary code (QR code).

[0170] In the case of a QR code with an icon like this, the image analysis unit 223 extracts the object Without determining whether it is an icon or not, it is determined to be an icon and recorded in the QR code. The auxiliary information provided is retrieved. Then, subsequent processing is carried out according to the auxiliary information.

[0171] Figure 13(b) shows an example of a QR code 532 with payment-related icons. The code section indicates that the icon shape is payment-related, and also lists the available payment methods. It will be recorded.

[0172] The image analysis unit 223 acquires this information by analyzing the image. Access icon DB231 and extract all icon shapes whose type 231c is payment-related. Then, as shown in Figure 13(c), it is displayed on the display 215. The following process is as described above. This is the same as the embodiment.

[0173] While this explanation uses payment-related icons as examples, it is not limited to these. No. It can be applied to various services and processes, such as transportation, commerce, and government services. can.

[0174] <Modification 9> In the above embodiment and modified example, the HMD200 itself is equipped with icon DB231. The image analysis unit 223 refers to the icon DB 231, and the extracted object is an icon. It determines whether or not it is. Similarly, for applications, HMD20 0 itself has an application DB232, and according to the application storage location 232d of the application DB232 The application is being launched.

[0175] However, the HMD200 itself is equipped with icon DB231 and application DB232. It is not necessary. Instead of icon DB231, server 300 will display various icon shapes. It may also have a registered icon bank 311. Also, without an app DB 232, You may also configure it to always retrieve applications from server 300.

[0176] In this case, as shown in Figure 14, each icon registered in the icon bank 311 is A It comprises icon ID 311a, icon shape 311b, type 311c, and content 311d. Furthermore, the icon contains the URL information of the application corresponding to that icon. The application URL (311e) and related information (311f) are registered.

[0177] Category 311c, like the icon DB231 mentioned above, is information classified by icon type. For example, transportation systems, payment systems, etc. Icon ID 311a uniquely identifies each icon. This is separate information. The corresponding app URL 311e is the app case associated with the icon. This is access information for the icon. Related information includes, for example, information about the source of the icon's registration and rights. This is information about the interest rates, etc.

[0178] Furthermore, the server 300 to which icon bank 311 is registered is the server to which the application is registered. It may be a different device from server 300.

[0179] In this case, the image analysis unit 223, when performing the icon identification process in step S1103, Access the icon bank 311 of server 300 via communication I / F217 and perform a query. Yes, it is identified. Then, the image analysis unit 223 identifies the icon ID 311a of the identified icon. Download icon shape 311b to HMD200, and icon shape 311b Display 215 will show the options and accept the selection. Note that when downloading, use the icon. Other information associated with the app (Type 311c, Content 311d, Corresponding app URL 311) Download e, etc., at the same time.

[0180] The instruction reception display unit 221 notifies the application acquisition unit 225 of the information of the selected icon. The app acquisition unit 225 then retrieves the corresponding app U associated with the selected icon. Follow the instructions in RL311e to access the URL and download the corresponding application. And then it starts up.

[0181] When determining whether an extracted object is an icon, only Icon DB231 is used. If the matching range is set to this, the processing speed is fast. On the other hand, by using icon bank 311 This will involve accessing server 300 via network 101, and the overall process The processing speed will be slower, but the likelihood of it being identified as an icon will increase.

[0182] In this way, you can register various icons and their related information in Icon Bank 311. Therefore, when an icon is used for the first time, it is recognized and the associated application is used. It can be used. Therefore, convenience is further improved.

[0183] The application itself may also be registered in Icon Bank 311.

[0184] Furthermore, the above embodiment may be combined with this modified example. That is, the image analysis unit 22 Step 3, after extracting the object, first accesses Icon DB231. Then, Icon If an icon corresponding to the extracted object is not registered in DB231, The icon bank 311 of the -bar 300 is checked. Then, the image analysis unit 223 checks the icon If you download icon data, etc., from Bank 311, then Icon DB231 Add to registration.

[0185] This means that even if the icon database 231 is not complete, you can use image 233 to access it. If we can extract objects that are similar to the icon in the world, then we can create a corresponding icon. It can launch the attached application. In other words, it can launch applications that are not widely used in general. Regarding this feature, prior registration will no longer be required, further improving convenience.

[0186] Note that if icon shape 311b that matches is not registered in icon bank 311, In conclusion, the object extracted from image 233 was determined to be one for which no similar icons exist. The system may then display a message to that effect on display 215 and interrupt the process. Also, user 201 You may notify them of this.

[0187] By using Icon Bank 311, various objects that exist in the real world can be represented. It can be used by registering it as an icon and assigning meaning or categorizing it. Even for objects that have not been attached or otherwise modified, the shape of the object, its placement, etc. It is also possible to derive meaning from this by analogy.

[0188] <<Second Embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, the display is HMD200 Although it is being done on display 215, in this embodiment it is further linked with a mobile terminal and the display is This is done on the display of a mobile device.

[0189] In this embodiment, the HMD200 communicates with the user 201's mobile device via WiFi. It uses wireless communication such as Bluetooth (registered trademark) and identifies the A with HMD200. It sends icon graphics, initial screen data of launched applications, etc., to the mobile device. On mobile devices, the received icon shapes and initial screens are displayed on the mobile device's screen. ru.

[0190] First, let's describe the system configuration of the application control system 100a of this embodiment. To clarify, Figure 15(a) shows the system of the application control system 100a of this embodiment. This is a diagram of the configuration. The application control system 100a of this embodiment is HMD200 It also includes a server 300 and a mobile terminal 400.

[0191] Similar to the first embodiment, the HMD200 and server300 are connected via network 101. They are connected in this manner. Furthermore, these configurations and functions are the same as those in the first embodiment.

[0192] However, as described above, in this embodiment, the HMD200 and the mobile terminal 400 are at close range. Data is transmitted and received wirelessly. Therefore, the communication I / F217 of this embodiment is a LAN (Local Area Network) communication interface and short-range wireless communication interface, Prepare.

[0193] The LAN communication interface is accessed via wireless connection such as Wi-Fi (registered trademark). Connected to the network via an AP (Application Platform) device, and data is exchanged with other devices on the network. This is an interface for sending and receiving data.

[0194] A short-range wireless communication interface (N / F) transmits and receives data wirelessly with other devices equipped with wireless communication equipment. To communicate. For example, information devices at a distance of several meters to tens of meters using Bluetooth, etc. It is an interface that uses radio waves to enable simple information exchange between devices. The interface is, for example, for Near Field Communication (NFC). It is an I / F of ation)) and is an NF at extremely short distances of a few centimeters to about 1 meter. Bidirectional communication between devices equipped with a C chip may be realized. For example, it may be installed in the body of a mobile terminal. It supports services that utilize contactless IC chips, such as electronic money.

[0195] Furthermore, the instruction reception display unit 221 receives instructions from the mobile terminal 400 via the communication I / F 217. I will accept instructions.

[0196] The mobile terminal 400 includes, for example, a smartphone or tablet, and has communication and display functions. This is an information processing device equipped with the following. The mobile terminal 400 may also be a wearable device such as a watch. Yes. When using wearable devices, the system can be controlled using only the wearable devices in combination with the HMD200. Because the system can be built, the advantage of not occupying the user's hands is maintained. In this embodiment, HMD2 00 means that the connection is made via short-range wireless communication. Short-range wireless communication includes, for example, Wi-Fi and Blu-ray. This includes things like Ethooth. Furthermore, the mobile terminal 400 is connected to network 101. It will be done.

[0197] Figure 15(b) shows the hardware configuration diagram of the mobile terminal 400 of this embodiment. The 400 basically has the same configuration as the HMD200 shown in Figure 2. That is, CPU4 11, RAM 412, ROM 413, camera 414 (outward-facing camera 414a and visual Line detection camera 414b), display 415, and audio interface (I / F) 41 6, communication I / F 417, and sensor 418 (GPS receiver 418a, gyro sensor 41 8b, accelerometer 418c, geomagnetic sensor 418d, touch sensor 418e), and each part It includes a bus 419 that connects to it. The configuration with the same name as the HMD200 shown in Figure 2 is similar to that of a machine. To possess the ability.

[0198] In other words, like the communication interface 217, the communication interface 417 also has a LAN communication interface and a short-range It is equipped with a wireless communication I / F. Furthermore, the communication I / F 417 is a mobile telephone communication network. A telephone network communication interface that transmits and receives calls and data via wireless communication with a base station, and USB The HMD200 can be connected via wired means such as Universal Serial Bus. A priority communication interface for sending and receiving data with other nearby devices, and other devices equipped with proximity radio communication devices. A proximity wireless communication interface (I / F) enables data transmission and reception between devices via wireless communication, and also enables infrared communication. It may also be equipped with an infrared communication interface.

[0199] In this embodiment, in the first embodiment, the display 215 of the HMD200 is displayed The data to be displayed is shown on display 415 instead of display 215.

[0200] Furthermore, user instructions received via display 415 are transmitted via communication I / F 417 The signal is transmitted to the HMD200 via the communication I / F217 and to the instruction reception display unit 221. It will be sent to [destination].

[0201] Below are examples of displays in the display 415 of this embodiment, shown in Figures 16(a) to 16(c) Let's explain with a concrete example using ). For example, when user 201 is near a taxi stand. In the first embodiment, the search instruction button 711 displayed on the display 215 is selected. In this embodiment, as shown in Figure 16(a), the display 415 has a search instruction button. Display 421. The display is performed by the CPU 411 of the mobile terminal 400.

[0202] Then, when user 201 selects the search command button 421, CPU 411 selects The CPU411 then accepts the selection and generates a signal indicating that the selection has been accepted. The output is sent to the HMD200 via the communication interface 417. The HMD200 uses the communication interface 2 This signal is received via 17 and passed on to the instruction reception display unit 221.

[0203] The instruction reception display unit 221 instructs the image analysis unit 223 to perform the same shooting process as in the first embodiment. The image is to be acquired by unit 222. Then, the image analysis unit 223 is to extract objects. The icon is identified. Then, once the image analysis unit 223 identifies the icon, the identified If there are multiple icons, the icon shape is transmitted to the mobile terminal 40 via the communication I / F 217. Send to 0.

[0204] On the mobile terminal 400 side, as shown in Figure 16(b), the CPU 411 receives the icon. The figure 422 is displayed on the display 415. Then, the CPU 411 receives information from the user. Once the selection is accepted, the accepted selection is sent to the HMD200 via the communication I / F417. Send.

[0205] On the HMD200 side, the instruction reception display unit 221 receives the selected content via the communication I / F 217. Upon receiving the message, the selection is sent to the application launch unit 224. This accesses the application DB232 in the same manner as in the first embodiment, and the application It determines whether or not the application is registered. Then, the application launch unit 224 determines whether the application is registered. If present, the application is launched. Then, the instruction reception display unit 221 displays the initial image. The surface data is transmitted to the mobile terminal 400 via the communication interface 217.

[0206] On the mobile terminal 400 side, as shown in Figure 16(c), the CPU 411 receives the initial image. The surface 423 is displayed on the display 415.

[0207] On the other hand, if no application is registered, the application acquisition unit 225 will contact the server 3 After obtaining the application from 00 and performing the same processing as in the first embodiment, the obtained application case The system is launched, and the startup screen data is sent.

[0208] According to this embodiment, the system interacts with a mobile terminal 400 held by the user 201. User 201 can then issue operating instructions using the mobile device 400 that they have in their possession. It's easy to use.

[0209] Note that various variations of the display mode on display 215 are shown in the display of this embodiment. This can also be applied.

[0210] For example, the display of the selected icon shape may differ from that of other unselected icon shapes. This includes overlaying icon shapes onto the acquired image. When displaying icon shapes overlaid on top of each other, the acquired image is also sent to the mobile terminal 400. .

[0211] Furthermore, various modifications of the first embodiment, other than the display configuration, can also be applied to this embodiment.

[0212] <<Third Embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the first embodiment or the The system configuration of the second embodiment is further enhanced by the addition of beacons. The following describes the first embodiment with beacons. This embodiment will be explained using the case where a controller is added as an example.

[0213] Figure 17(a) shows the system configuration of the application control system 100b of this embodiment. This is a diagram. The application control system 100b of this embodiment includes an HMD200 and a It is equipped with a 300-bar and a 800-beacon.

[0214] The Beacon 800 outputs pre-configured unique ID information. For example, BLE (Bl A signal conforming to the Bluetooth Low Energy specification (hereinafter referred to as a beacon signal) It transmits continuously. This beacon signal is activated when an application that receives the beacon signal is activated. It is only received on the device that is set to receive it.

[0215] The beacon 800 in this embodiment is a transmitter that emits a beacon signal. 0 outputs a beacon signal once every few seconds. This beacon signal is, for example, within a radius of several seconds. The output should be such that it reaches a range of 10 meters.

[0216] Similar to the first embodiment, the HMD200 and server300 are connected via network 101. They are connected in this manner. Furthermore, these configurations and functions are essentially the same as those of the first embodiment.

[0217] However, the communication I / F217 of the HDM200 in this embodiment receives beacons from the beacon 800. It also functions as a signal receiving unit that receives BLE signals. In other words, it can receive BLE-compliant signals. It is possible.

[0218] Furthermore, the HMD200 will enter search mode when it receives a signal from the Beacon 800. To achieve this, the HMD200 of this embodiment is as shown in Figure 17(b), first In addition to the functions of the embodiment, the beacon receiving unit 228 that performs beacon signal reception processing is further Prepare.

[0219] When the beacon receiver 228 receives a beacon signal from the beacon 800, it receives the instruction. A search command is sent to the display unit 221.

[0220] As mentioned above, the beacon signal is used by applications to perform processing in accordance with this beacon signal. It is received only on devices equipped with the beacon receiving app (hereinafter referred to as the "beacon receiving app"). In this configuration, the program for this beacon receiving application is registered in the storage device 230. Then, upon startup of the HMD200, or according to instructions from user 201, CP U211 loads and runs the beacon receiving application in RAM212, thereby enabling beacon reception. A communication interface 217 and a beacon receiver 228 are implemented to receive the beacon signal. The controller receiving app runs in the background at all times after launch.

[0221] Thus, in this embodiment, the HMD200 is located within an area where it can receive beacon signals. Upon entering, the HMD200 receives a beacon signal via the communication I / F217. Then, The received beacon signal is transmitted to the beacon receiver 228, where it is treated as a search instruction and the instruction is accepted. This is transmitted to the display unit 221.

[0222] The instruction reception display unit 221, upon receiving a search instruction, processes the image processing unit 22, as in the first embodiment. In step 2, the camera starts taking images, that is, acquiring image 233. Then, the image analysis unit 223 analyzes the image Each time an image 233 is acquired, the image 233 is analyzed to extract objects and icons. The application launch unit 224 then launches the application with the specified icon. do.

[0223] As an example, the Beacon 800 is used in shops, restaurants, etc., as shown in Figure 18(a). Let's take the example of when it is placed in the accounting area. In this case, the Beacon 800 is located in the accounting area. It is placed near the cash register. Near the cash register, there are usually available credits. Card logos and logos indicating usable IC cards are affixed to the card.

[0224] When user 201 approaches the register, HMD200 receives a beacon signal from beacon 800. The beacon signal is received. The instruction reception display unit 221 receives the beacon signal via the communication I / F 217. Then, the image processing unit 222 is instructed to acquire an image within the field of view. Then, the image analysis unit 22 Step 3 extracts the object and identifies the icon.

[0225] In the accounting area, user 201, through the transparent display 215, for example, Figure Observe a scene (real image) 550 as shown in 18(b). Specifically, near the cash register, the store The sight of stickers indicating various credit cards and electronic money accepted there. Yes. Here, upon receiving the beacon signal, the imaging processing unit 222 performs the following actions: Obtain image 650 as shown.

[0226] The image analysis unit 223 extracts a group of objects from this image 650. An example of the object group 620 is shown in Figure 19(a). The image analysis unit 223 then analyzes the icons Refer to DB231 to identify the object that is the icon.

[0227] The instruction reception display unit 221 then displays the icon shape of the object identified as an icon. Extract 231b from icon DB231 and display it on display 215. (See example display) This is shown in Figure 19(b). Note that here, the user 20 is shown through the transparent display 215. The external scenery that comes into view of point 1 will be omitted. The same applies to subsequent points.

[0228] The instruction reception display unit 221 receives selection instructions from the user 201. For example, part 221 distinguishes the selected icon shape 231b from other icon shapes. The shapes may be displayed in a specific manner. For example, as shown in Figure 19(c), the selected icon shape Display it larger than other icon shapes.

[0229] The application launch unit 224 is registered in the application DB 232 in association with the selected icon. Launch the application that is running. Here, for example, as shown in Figure 19(d), A screen for making payments using a card, represented by an icon, is displayed on display 215.

[0230] If an application is not registered to correspond to the selected icon, Similar to the first embodiment, the application acquisition unit 225 acquires the application from the server 300.

[0231] Thus, in this embodiment, the neighborhood where an object corresponding to an icon of the external world is placed Place a beacon there. And instead of turning on exploration mode, use Beacon 800 or When these signals are received, the HMD200 starts application control processing.

[0232] Therefore, according to this embodiment, the same effects as the first embodiment are achieved. Depending on the implementation method, operations such as turning on the search mode become unnecessary, and the user 201 The gaps between stops will be reduced, and convenience will increase.

[0233] <Variation 10> In this embodiment, the beacon signal output from beacon 800 is basically A This signal indicates that an object equivalent to an icon is nearby. However, The beacon signal transmitted by Beacon 800 may include information other than location information. Example For example, this includes information that limits the processing performed in the vicinity.

[0234] This allows the image analysis unit 223 to limit the types of icons 231c that it matches. This reduces the annoyance of frequent reactions to external markers.

[0235] For example, near locations where payments are made, only icons related to payments are identified. In addition, the instruction reception display unit 221, upon receiving a beacon signal, not only outputs a search instruction, but also The beacon signal is also output to the image analysis unit 223.

[0236] The image analysis unit 223 analyzes the beacon signal and selects the required icon type 231c. Determine. Then, when identifying the icon corresponding to the extracted object, the identified type Match only icons that match 231c.

[0237] For example, register the beacon signal to extract only icons where type 231c is payment. If so, in the example in Figure 19(b), from left to right, the icons registered in the icon DB are The icon type is payment, payment, payment, payment, point accrual. In this case, the rightmost one is A The icon is not displayed on display 215.

[0238] In this embodiment as well, as in the second embodiment, the display destination is the mobile terminal 400. It's okay to have it.

[0239] <Variation 11> The HMD200 in each of the above embodiments and modified examples is further equipped with a user authentication function. That's good too.

[0240] The user authentication function confirms that user 201 of the HMD200 is a pre-registered user 201. Authenticate the user's actions and instruct the application to perform the above application control processing only if the authentication is successful. This is a security feature that demonstrates its capabilities.

[0241] Figure 20 shows the functional block diagram of the HMD200 in this modified example. App control unit 2 of this modified example 20 is a function of the first embodiment, further used to authenticate the user 201 of the HMD200. It is equipped with an authentication unit 229 for user authentication.

[0242] For user authentication, for example, iris recognition can be used. In this case, the user authentication device This functionality is achieved, for example, by the authentication unit 229 and the gaze detection camera 214b.

[0243] In iris authentication, the iris of user 201 is photographed in advance, and the matching data 236 is stored in ROM 213. It is then stored in memory. When user 201 puts on the HMD200, the authentication unit 229 performs eye-tracking Camera 214b acquires an image of user 201's iris and compares it with a previously stored iris image. The authentication unit 229 then, if the verification is successful, confirms that the user 201 is legitimate. The instruction reception display unit 221 is notified when it receives notification of successful verification. In this case, the search command button is displayed on the display 215.

[0244] Note that user authentication is not limited to the methods described above. For example, by tracking the eye gaze of the authentication pattern. Unlocking, the number of blinks in a predetermined time or blinking actions at predetermined time intervals, or finger It may also be a crest authentication function, etc. Furthermore, a combination of these may also be used. External authentication methods may also be used. The authentication pattern is, for example, displayed on the instruction reception display unit 221. Create from this.

[0245] As a result, it is possible to prevent fraudulent transactions, such as payment processing, by preventing others from impersonating you. Therefore, the HMD200 achieves high levels of safety while offering convenience and ease of operation. can.

[0246] <Variation 12> In the first embodiment, the third embodiment, and each of the modified examples, the display terminal is an HMD. Although the example of using 200 was used, the display terminal is not limited to this. In this example, mobile devices such as smartphones and tablet devices equipped with cameras and communication functions... It's okay to have it.

[0247] In any case, simply taking a picture of a real object that resembles an icon will create an app. You can launch the application. Therefore, the icon of the application you want to use This eliminates the need to search for and launch the application from the display screen, improving usability. do.

[0248] In addition, in the first and second embodiments described above, when the search mode is turned ON, We have acquired images of the external environment (233) and started image analysis, but this is not limited to that. For example, Images of the outside world 233 are acquired at predetermined time intervals, and each time an image is acquired, the image analysis unit 2 23 may be configured to identify the icon.

[0249] In this case, the display icon 71 that can be selected at that time is always displayed on the display 215. 2 is displayed. User 201 can run the application at the time the display appears. You can simply select from the icons 712.

[0250] Furthermore, the present invention is not limited to the embodiments and modifications described above, and various modifications are possible. Examples are included. For example, the embodiments and modifications described above clearly illustrate the present invention. This is a detailed explanation, and is not necessarily limited to those that possess all the configurations described. This does not mean that a part of the configuration of one embodiment or modified example may be used in other embodiments or modified examples. It is possible to replace the configuration with that. Also, in one embodiment or modified configuration, other embodiments It is also possible to add configurations of different embodiments or variations. Furthermore, each embodiment or variation It is possible to add, delete, or replace parts of the configuration with other components.

[0251] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be partially or entirely modified, for example. It may also be implemented in hardware by designing it with an integrated circuit. These processes occur when the processor interprets and executes programs that perform their respective functions. This can also be implemented in software. This includes programs, tables, files, etc., that implement each function. The information is located in the memory section, hard disk, and SSD (Solid State Drive). It can be stored on recording devices such as IC cards, SD cards, DVDs, etc. ru.

[0252] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes and do not necessarily represent all of them in the actual product. This does not necessarily indicate control lines or information lines. In reality, almost all components are interconnected. It is reasonable to assume that this is the case. [Explanation of symbols]

[0253] 100: Application control system, 100a: Application control system, 100b: Application control system, 101: Network, 200: HMD, 201: User, 211: CPU, 212: RAM, 213: ROM ,214: Camera, 214a: Outward-facing camera, 214b: Eye-tracking camera, 215: Display PLAY, 216: Audio I / F, 217: Communication I / F, 218: Sensor, 218a: GPS Receiver, 218b: Gyroscope sensor, 218c: Accelerometer, 218d: Geomagnetic sensor 218e: Touch sensor, 219: Bus, 220: Application control unit, 221: Instruction reception sheet Display unit, 222: Image processing unit, 223: Image analysis unit, 224: Application launch unit, 225: App 228: Beacon acquisition unit, 229: Authentication unit, 230: Storage device, 231: Icon DB, 231a: Icon ID, 231b: Icon shape, 231c :Type, 231d:Content, 232:App DB, 232a:Icon ID, 232b:A App name, 232c: App ID, 232d: App storage location, 233: Image, 234: Display data, 235: Store DB, 236: Matching data, 241 : Display control unit, 242: Display image generation unit, 243: Eye-gaze detection unit, 244: Analysis control unit, 300: Server, 311: Icon Bank, 311a: Icon ID, 311b: Eye Conformation, 311c: Type, 311d: Content, 311e: Compatible app URL, 311f: Related information, 400: Mobile device, 411: CPU, 412: RAM, 413: ROM, 414: Camera R, 414a: outward-facing camera, 414b: gaze detection camera, 415: display, 417 :Communication I / F, 418:Sensor, 418a:GPS receiver, 418b:Gyro sensor, 418c: Accelerometer, 418d: Geomagnetic sensor, 418e: Touch sensor, 419: Bus, 421: Search command button, 422: Icon shape, 423: Initial screen, 501: Signboard, 510: Actual image, 511: Bus diagram, 512: Train diagram, 513: Ta Kushi's shapes: 514: Person, 515: Flower, 516: Weed, 517: Dog, 518: Mark 531: QR code, 532: QR code with icon, 541: receipt, 550: actual image, 601: Image, 602: Object, 603: Taxi dispatch app, 610: Image, 611: Shape, 612: Shape, 613: Shape, 614: Shape, 615: Shape, 616: Figure Shapes, 617: Figures, 618: Circles, 619: Ellipses, 620: Groups of Objects, 621: Characters Column, 622: string, 623: string, 650: image, 711: Search command button, 712: Display icon, 713: Initial screen data, 714: App icon, 715: Instruction button, 800: Beacon

Claims

1. Display terminal, The photography department, Display unit and Control unit and Memory device and Equipped with a communication device, The display terminal has a search mode, The control unit, When the user switches to the search mode based on their instructions, the imaging unit is controlled to acquire an image within the user's field of view. The system controls the analysis to determine whether or not an icon shape corresponding to an executable application is present in the image. If the storage device does not contain the application corresponding to the icon shape, the device is controlled to acquire the application via the communication device. If the storage device does not contain the icon shape corresponding to the application, the device is controlled to acquire the icon shape via the communication device. Display terminal.

2. A display terminal according to claim 1, The control unit, The process of performing the analysis and launching the application corresponding to the icon shape is repeated at predetermined time intervals. If the icon shape is displayed after the process of launching the application, the display unit is controlled so that the icon shape is not displayed. Display terminal.

3. A display terminal according to claim 1, The aforementioned display terminal is a head-mounted display. Display terminal.

4. A display terminal according to claim 1, The control unit controls the system to terminate the search mode based on a user's instruction to terminate the search mode. Display terminal.

5. A display terminal according to claim 1, The display unit displays a search instruction button, The control unit controls the system to switch to the search mode in response to the selection of the search instruction button. Display terminal.

6. A display terminal according to claim 1, The control unit controls the imaging unit to acquire an image of the user's field of view once when it switches to the search mode. Display terminal.

7. A display terminal according to claim 1, The control unit controls the display mode to change depending on whether the corresponding application is present in the storage device. Display terminal.

8. A display terminal according to claim 1, If there is no application corresponding to the icon shape in the storage device, the control unit controls itself to determine whether there are multiple applications corresponding to the icon shape. If there are multiple applications corresponding to the aforementioned icon shape, the display unit will display information about each of the multiple applications. The control unit controls the acquisition of an application based on the user's selection instruction for the plurality of applications. Display terminal.

9. A display terminal according to claim 1, If no application corresponding to the icon shape is found in the storage device, the display unit will display an application that is highly related to the application corresponding to the icon shape. Display terminal.

10. A method for controlling a display terminal, The aforementioned display terminal is If the system switches to search mode based on user instructions, the steps include acquiring an image of the user's field of view, The steps include: analyzing whether or not an icon shape corresponding to an executable application is present in the image; If the storage device provided in the display terminal does not contain the application corresponding to the icon shape, the step of obtaining the application is as follows: If the storage device does not contain the icon shape corresponding to the application, the following steps are performed: A method for controlling the display terminal.

11. A control method for a display terminal according to claim 10, The aforementioned display terminal is The process of performing the analysis and launching the application corresponding to the icon shape is repeated at predetermined time intervals, and after the process of launching the application, if the icon shape is displayed, the process of not displaying the icon shape is executed. A method for controlling the display terminal.

12. A control method for a display terminal according to claim 10, The aforementioned display terminal is a head-mounted display. A method for controlling the display terminal.

13. A control method for a display terminal according to claim 10, The display terminal performs the step of ending the search mode based on an instruction from the user to end the search mode. A method for controlling the display terminal.

14. A control method for a display terminal according to claim 10, The display terminal performs the steps of displaying a search instruction button, The following steps are performed in response to the selection of the search instruction button: switching to the search mode; A method for controlling the display terminal.

15. A control method for a display terminal according to claim 10, The display terminal, upon switching to the search mode, once performs the step of acquiring an image of the user's field of view. A method for controlling the display terminal.

16. A control method for a display terminal according to claim 10, The display terminal performs a step of changing the display mode depending on whether a corresponding application is present on the display terminal. A method for controlling the display terminal.

17. A control method for a display terminal according to claim 10, The display terminal, if there is no application corresponding to the icon shape on the display terminal, takes the step of determining whether there are multiple applications corresponding to the icon shape. If there are multiple applications corresponding to the aforementioned icon shape, the step is to display information about each of the multiple applications, The steps of obtaining an application based on the user's selection instructions for the aforementioned multiple applications are performed. A method for controlling the display terminal.

18. A control method for a display terminal according to claim 10, The aforementioned display terminal is If the application corresponding to the icon shape is not available on the display terminal, the step of displaying an application highly related to the application corresponding to the icon shape is executed. A method for controlling the display terminal.

19. A display terminal, The photography department, Display unit and Control unit and Memory device and Equipped with a communication device, The display terminal has a search mode, The control unit, When the user switches to the search mode based on their instructions, the imaging unit is controlled to acquire an image within the user's field of view. The system controls the analysis to determine whether or not an icon shape corresponding to an executable application is present in the image. If the application corresponding to the aforementioned icon shape is not found in the storage device, the system controls the acquisition of the application via the communication device. The process of performing the analysis and launching the application corresponding to the icon shape is repeated at predetermined time intervals. If the icon shape is displayed after the process of launching the application, the display unit is controlled so that the icon shape is not displayed. Display terminal.

20. A method for controlling a display terminal, The aforementioned display terminal is If the system switches to search mode based on user instructions, the steps include acquiring an image of the user's field of view, The steps include: analyzing whether or not an icon shape corresponding to an executable application is present in the image; If the storage device provided in the display terminal does not contain an application corresponding to the icon shape, the step of obtaining the application is as follows: The process of performing the analysis and launching the application corresponding to the icon shape is repeated at predetermined time intervals, and after the process of launching the application, if the icon shape is displayed, the process of not displaying the icon shape on the display unit provided in the display terminal is performed. A method for controlling the display terminal.

Images