Display terminal and control method of display terminal

Abstract

To provide a display terminal having a small display area and achieving high operability while maintaining convenience.SOLUTION: High operability while maintaining convenience is achieved in a display terminal having a small display area. A display terminal 200 having a photographing device 214 is provided with an image analysis unit 223 for analyzing an image 233 photographed by the photographing device 214, and specifying an icon related to an actual object by referring to an icon DB 231 using area information, which is information of an area on the image 233 and corresponding to the actual object existing in the outside world, and an application starting unit 224 for acquiring an application associated with the specified icon from an application 232 and starts it.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display control technique for a display terminal with a small display area. [Background technology]

[0002] As a display terminal with a small display area, for example, a display device supported on the user's head and placed in front of the user's eyes HMD (Head Mount Display) for displaying images Among HMDs, there is a see-through type that allows users to see both the outside world and the display. (See-through type) has a particularly small display area. As a technique for improving the operability of operations using a display, for example, the technique disclosed in Patent Document 1 is There is a trick.

[0003] The HMD system disclosed in Patent Document 1 includes a coordinate system setting unit, a segment selection unit, and an image generation unit. The coordinate system setting unit sets a cylindrical seat divided into a plurality of segments in the real space. The segment selector sets the coordinate system based on the display orientation in the cylindrical coordinate system. The image generating unit sets one of the plurality of segments in a selection standby state. When the selected segment is confirmed, the display will show the segment according to the selected segment. Produces a display image shown. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2014 / 129105 Summary of the Invention [Problem to be solved by the invention]

[0005] The HMD is similar to commonly used information terminals such as personal computers and smartphones. Similarly, by placing and displaying icons on the display and accepting their selection, The application software associated with the selected icon is started.

[0006] In the technology disclosed in Patent Document 1, the real space around the HMD wearer is divided into multiple segments. Then, assign selectable objects (i.e. The icon displayed in the segment in the HMD wearer's line of sight is selected. This allows the user (HMD wearer) to hold the device in his / her hand and perform input operations. Operation instructions can be given without using any device, improving operability.

[0007] However, according to the technology disclosed in Patent Document 1, all applications stored in the HMD are The icons of the options are displayed, and you can select the one you want. Since segments for displaying each icon are provided at the cylindrical coordinates, the desired icon can be displayed To make a selection, the user must rotate his head. Also, the number of selectable icons If you have a large number of icons, the number of segments will also increase. As a result, the area of ​​each segment becomes smaller. The size of the icons is also smaller. Also, since a lot of space is used to display the icons, it is difficult to see the outside world. The visible area is also smaller. Therefore, the more applications you store, the This increases convenience but reduces operability.

[0008] The present invention has been made in view of the above circumstances, and is directed to providing a display terminal with a small display area that is convenient and easy to use. The objective is to provide a technology that realizes high operability while maintaining convenience. [Means for solving the problem]

[0009] The present invention relates to a photographing device, and a method for analyzing and applying an image photographed by the photographing device. A display terminal having an application control unit that executes In response to the instruction, the image capturing device captures an image of the user's field of view and transmits it to the application. The application control unit corresponds to an executable application displayed in the image in the field of view. An icon graphic that corresponds to the image is searched for based on the image in the visual field range, and the icon graphic is selected by the search. When one shape is found, the application corresponding to that icon shape is automatically launched. If a plurality of icons are found, the icon graphics are displayed on the display terminal, and the user The present invention is characterized in that an application corresponding to the selected icon graphic is started.

[0010] The present invention also relates to a display terminal, a server connected to the display terminal via a network, An application control system comprising: a photographing device; a storage device; and the display terminal In response to an instruction from the user, the image capturing device captures an image of the user's field of view, and The icon shapes corresponding to the executable applications displayed in the field images, An image analysis unit that searches based on the image in the field of view, and Determine whether an application associated with the con-figuration is stored in the storage device. and when it is determined that the application is not stored in the storage device, an acquisition unit that acquires the application from the server, and the launch unit further includes: If the number of the icon figures found by the search is one, The application selected by the user is automatically started, and if multiple applications are selected, The system is characterized by starting an application associated with an icon graphic.

[0011] The present invention also provides an application control method for a display terminal equipped with an image capture device. Then, according to an instruction from the user of the display terminal, an image in the visual field of the user is acquired, and the visual field of the user is displayed. The icon shapes corresponding to the executable applications displayed in the field images, A search is performed based on the image in the field of view, and one of the icon figures is found by the search. In the case of one, the application associated with the icon shape will be automatically started, and in the case of multiple displays the icon graphic on the display terminal, and the icon selected by the user is The method is characterized by starting an application associated with the graphic. Effect of the Invention

[0012] According to the present invention, in a display terminal with a small display area, it is possible to provide a display device with high operability while maintaining convenience. Problems, configurations and effects other than those described above will be solved by the following embodiment. It will be revealed. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1A is a system configuration diagram of an application control system according to a first embodiment, and FIGS. 1B to 1E are explanatory diagrams for explaining an overview of application control according to the first embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of an HMD according to the first embodiment. [Diagram 3] FIG. 4A is a functional block diagram of an application control unit according to the first embodiment, and FIG. 4B is a functional block diagram of a gaze instruction function of an instruction acceptance display unit according to the first embodiment. [Figure 4] FIG. 2A is an explanatory diagram for explaining an example of an application database according to the first embodiment, and FIG. 2B is an explanatory diagram for explaining an example of an icon database according to the first embodiment. [Diagram 5] 11 is a flowchart of an application control process according to the first embodiment. [Figure 6] 2A to 2C are explanatory diagrams illustrating an application control process according to the first embodiment. [Figure 7] 2A to 2C are explanatory diagrams illustrating an application control process according to the first embodiment. [Figure 8] FIG. 13 is an explanatory diagram for explaining a user instruction method in variant example 1 of the first embodiment, FIG. 13 is an explanatory diagram for explaining an example of icon display in variant example 2 of the first embodiment, and FIG. 13 is an explanatory diagram for explaining a user instruction method when launching an app in variant example 5 of the first embodiment. [Figure 9] 13A to 13C are explanatory diagrams for explaining objects in the fourth modified example of the first embodiment, and FIG. 13D to FIG. 13F are explanatory diagrams for explaining a display example after identifying an icon in the fourth modified example of the first embodiment, respectively. [Figure 10] FIG. 13A is an explanatory diagram for explaining an application database according to a sixth modified example of the first embodiment, and FIG. 13B is an explanatory diagram for explaining an example of a display according to this modified example. [Figure 11] FIG. 23 is an explanatory diagram for explaining an example of a store database according to a seventh modified example of the first embodiment. [Figure 12] 13(a) and 13(b) are explanatory diagrams for explaining an eighth modified example of the first embodiment. [Figure 13] 13(a) to 13(c) are explanatory diagrams illustrating an application control process according to an eighth modified example of the first embodiment. [Figure 14] FIG. 13 is an example of a system configuration diagram of a ninth modified example of the first embodiment. [Figure 15] FIG. 13A is a system configuration diagram of the second embodiment, and FIG. 13B is a hardware configuration diagram of a mobile terminal of the second embodiment. [Figure 16] 13(a) to 13(c) are diagrams illustrating examples of displays on a mobile terminal according to a second embodiment. [Figure 17] FIG. 13A is a system configuration diagram of a third embodiment, and FIG. 13B is a functional block diagram of an HMD of the third embodiment. [Figure 18] 13A to 13C are explanatory diagrams illustrating an application control process according to a third embodiment. [Figure 19] 13A to 13D are explanatory diagrams illustrating an application control process according to a third embodiment. [Figure 20] FIG. 23 is a functional block diagram of an HMD according to a twelfth modified example of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Unless otherwise specified, the same reference numerals are used for components having the same functions, and repeated explanations are omitted. It should be noted that the present invention is not limited to the embodiments described herein.

[0015] <<First embodiment>> In this embodiment, unnecessary icons are not displayed on the display terminal, and a desired application is displayed. The application control system provides a system that launches applications. Therefore, there is no need to search for an icon to launch a desired application among many applications.

[0016] Hereinafter, in this embodiment (this specification), a figure or symbol indicating a function of a program will be referred to as When the user selects and instructs execution, the program starts up. In this embodiment (this specification), the symbol is called an icon. A program executed by a line instruction is called an application or app.

[0017] In this embodiment, the display terminal is a head mounted display (HMD) that is worn by the user on the head. A head mounted display (HMD) that is connected to the outside world We will explain this using a see-through HMD as an example, which allows both the image and the display to be seen. do.

[0018] FIG. 1(a) is a system configuration diagram of an application control system 100 according to this embodiment. The application control system 100 of this embodiment includes an HMD 200 and a server 3. The HMD 200 and the server 300 communicate with each other via a network 101. Sends and receives data.

[0019] The server 300 of this embodiment is installed by a company that provides application-related services. In this embodiment, the server device analyzes the request data transmitted from the HMD 200. The server 300 analyzes the information and provides the relevant service to the requesting HMD 200. It stores various applications and accesses the HMD200 according to the request from the HMD200. The server 300 is, for example, an app store.

[0020] In this figure, the HMD 200 and the server 300 connected to the network 101 are Although only one of each is shown, the number of these is not important. and may transmit and receive data between them.

[0021] First, application control by the application control system 100 of this embodiment The outline of this will be explained with reference to Figs. 1(b) to 1(e).

[0022] As shown in FIG. 1B, a user 201 wearing an HMD 200 reads a signboard 501 or the like. If you find a shape that seems to be an icon, turn on the search mode of the HMD200. The search mode will be described later. At this time, the user 201 The figure is included in the photographing area displayed on the display of the Adjust the distance from the object, the orientation of the object relative to the shape, etc.

[0023] In response to the search mode being turned on, the HMD 200 displays an image in the visual field of the user 201. An example of the image 601 acquired here is shown in FIG. 1(c). to indicate a taxi stand.

[0024] Then, the HMD 200 analyzes the acquired image 601 and extracts the shape or the like corresponding to the actual field of view. An area (hereinafter, called an object) 602 on an image 601 is extracted. In the figure, the dotted rectangular area is the extracted object 602. Details of the extraction will be described later. .

[0025] When the HMD 200 extracts the object 602, The icon that corresponds to the specified icon is determined to be selected, and the specified icon is displayed as shown in Fig. 1(e). The application stored in association with the application is started. In this example, the taxi dispatch application 603 is started. The user 201 selects a desired process from the list and executes it.

[0026] Previously, the HMD200 stored icons for all available applications. The icon you need at that time is selected and executed. According to the embodiment, images of real objects in the field of view of the user 201 are displayed as icons. The application is then automatically started. There is no need to search for and select the desired icon from among the icons, and it is as if it exists in the surrounding environment. It's as if the actual shapes were used instead of icons to quickly find related applications you need. It can launch the application.

[0027] The HMD 200 of this embodiment that realizes the above processing will be described below.

[0028] [Hardware configuration] FIG. 2 is a hardware configuration diagram of the HMD 200 of this embodiment. 200 basically has the same configuration as a general-purpose computer (information processing device). As shown in FIG. 2, the HMD 200 includes a CPU 211, a RAM 212, and a ROM 213. 3, a camera 214, a display 215, and an audio interface (I / F) 216. , a communication interface (I / F) 217, a sensor 218, and a bus 219 connecting each part. And, it is equipped with.

[0029] The camera 214, which is a photographing device, includes an outward-facing camera 214a and a line-of-sight detection camera 214b. The outward camera 214a is configured to include the field of view of the user 201 of the HMD 200 in its imaging range. The gaze detection camera 214b is a camera that detects the gaze direction of the user 201. For this reason, the gaze detection camera 214b captures the iris, pupil, etc. of the user 201's eye. It can be attached so that it is included in the shadow range.

[0030] The display 215 displays images captured by the camera 214 and images generated within the HMD 200. The display 215 displays the surface data. For example, the display 215 is a transmissive liquid crystal device or an organic electroluminescence (EL) device. Devices, or MEMS (micro electro mechanical systems) However, the device is composed of an optical scanning device using a However, the present invention is not limited to this, and the direction of the display 215 may be changed while displaying an image on the display 215. Any device that can realize a structure that allows the side to be seen through will suffice.

[0031] The audio I / F 216 is, for example, a microphone, a speaker, a buzzer, etc. , sounds created in the HMD 200 and sounds such as voice and music sent via the communication I / F 217 In this embodiment, the audio I / F 216 is not necessarily provided.

[0032] The communication I / F 217 includes an encoding circuit, a decoding circuit, an antenna, and the like, and is connected to the network 101. In this embodiment, data is transmitted and received (data communication) with other devices via the via an access point or a base station of a mobile telephone network (not shown) This is an I / F that connects to the network 101 via the communication I / F 217. The MD 200 transmits and receives data to and from each server 300 connected to the network 101. .

[0033] The connection between the HMD 200 and the access point is, for example, Wi-Fi (registered trademark) or the like. The communication is performed by the wireless communication method of the above or other communication methods. The connection to the base station of the network is, for example, W-CDMA (registered trademark) (Wideband Code Division Multiple Access). e Division Multiple Access) system and GSM (registered trademark) Global System for Mobile communications) system, LTE (Long Term Evolution) system, or other communication systems This is done by the formula.

[0034] The sensor 218 detects the current position, inclination, speed, and operation by the user 201 of the HMD 200. The sensor 218 is, for example, a position information acquisition sensor such as a GPS receiver 218a. (position information acquisition device), gyro sensor 218b, acceleration sensor 218c, geomagnetic sensor 218d, a touch sensor 218e, etc. Note that the sensor 218 does not necessarily have to include all of the components. Good too.

[0035] In addition, the RAM 212 and the ROM 213 are collectively referred to as the RAM 212 and the ROM 213 unless there is a particular need to distinguish them. These are collectively referred to as a storage device 230. Also, the audio I / F 216 does not necessarily have to be provided.

[0036] [Function block] FIG. 3(a) shows the application control process of the HMD 200 according to the present embodiment. A functional block diagram of an application control unit (hereinafter referred to as an application control unit) 220, which is a function of The application control unit 220 includes an instruction acceptance display unit 221, an image capture processing unit 222, and an image resolution unit 223. The application software includes an analysis unit 223, an application launch unit 224, and an application acquisition unit 225.

[0037] Each of these units is configured such that the CPU 211 executes the programs stored in the storage device 230 in the RAM This is realized by loading the program into a memory such as 212 and executing it.

[0038] The storage device 230 also includes an icon database (icon DB) 231 and an application An application database (application DB) 232, an image 233, and display data 234, is stored.

[0039] The application DB 232 stores icons and applications that are started by selecting the icons. This is a database that registers applications in association with each other. An example of 32 is shown below.

[0040] As shown in the figure, the application DB 232 stores the application information associated with each icon. In this embodiment, the application information is registered. Icon ID, which is information that uniquely identifies an icon (referred to as an icon or graphic below) 232a, the name of the application (app name) 232b, The application ID 232c, which is information that uniquely identifies the application, The application storage location 232d, which is information, is registered. 00 in the storage device. If the application is not stored in the same location as the application on the network, It may be a URL that is the location of the server 300 that is connected to the server.

[0041] The icon DB 231 is a database for registering icons. Here is an example.

[0042] As shown in the figure, the icons are respectively icon ID 231a and icon shape 2 31b, type (category) 231c, and content 231d. The icon graphic 231b is information that uniquely identifies the icon. This is shape information of the figure to be displayed on the display 215.

[0043] The type 231c is preset for each process of the application associated with the icon. In this embodiment, for example, the categories of transportation, payment, facilities, products, shopping, etc. are registered. , games, music and video streaming, etc.

[0044] The content 231d corresponds to the description and / or icon of the icon graphic 231b. The outline of the process of the application is registered in the content 231d. This is used to analyze the meaning of a captured image and identify related icons, as described below.

[0045] The storage device 230 also stores an image 233 acquired by the image processing unit 222. In addition, the instruction reception display unit 221 generates screen data to be displayed on the display 215. Various data used when displaying the image is stored as display data 234.

[0046] The instruction reception display unit 221 receives operation instructions from the user 201. Screen data is generated from the processing results and displayed on the display 215.

[0047] The instruction reception display unit 221 displays, for example, various operation instruction buttons (areas ) and accepts the selection by the user 201. The data of the operation instruction button is stored in advance. The information is stored in a memory device.

[0048] The user 201 gives an instruction such as a selection by, for example, a line of sight. The direction of the user's gaze is detected and used as a substitute for a pointing device. The area on 215 that is in the line of sight is determined to be selected.

[0049] The gaze direction is detected using existing technology. For example, the gaze direction is detected based on the fixed reference point of the eye of the user 201. The fixed reference point is detected based on the position of the moving point. For example, the inner corner of the eye is used. The moving points are, for example, the pupil, the iris, etc. These positions are detected by the line of sight detection camera 2. The selection instruction is obtained by analyzing the image captured by the camera 14b. A predetermined number of blinks may also be added.

[0050] Here, the function block of the instruction reception display unit 221, which mainly realizes instructions by gaze, is As shown in this figure, the instruction acceptance display unit 221 is 1, a display image generating unit 242, a gaze detecting unit 243, and an analysis control unit 244.

[0051] The display image generating unit 242 generates screen data to be displayed on the display 215. The data is generated, for example, using various data stored in the storage device 230.

[0052] The display control unit 241 transmits the screen data generated by the display image generating unit 242 to the display 21. Display it at 5.

[0053] The gaze detection unit 243 detects the gaze direction from the image captured by the gaze detection camera 214b. .

[0054] The analysis control unit 244 analyzes the gaze direction detected by the gaze detection unit 243, identifies an instruction, If the line of sight direction remains the same for a predetermined period of time, the analysis control unit 244 outputs a signal to each unit. The image area present in the line of sight is determined to be selected. If the blinks are in the same direction and occur a predetermined number of times at a predetermined interval, the image area is selected. It may be determined that.

[0055] The operation instruction may include, for example, an instruction to set the processing of the HMD 200 to a search mode (search instruction ), and the selection of desired icons or applications from the extracted icons An instruction to execute a selected application, etc. When the unit 221 receives each instruction, it outputs a selection instruction signal and an execution instruction signal, and output.

[0056] The search mode is a mode for capturing an image of the actual field of view of the user 201. When the search instruction is received, the image acquisition unit 221 outputs the image acquisition instruction to the image acquisition unit 222. The shooting instruction signal is output until the search mode end instruction is received. .

[0057] The search instruction is given, for example, by the user 20 pressing a search instruction button displayed on the display 215. The search instruction button is selected by the user, for example, when the HMD 200 is started. After this, or after receiving an instruction to start the application control process, Display 2 It will be displayed at 15.

[0058] The selection instruction is, for example, to select a desired icon from among a plurality of icons displayed on the display 215. The execution instruction is an instruction to execute an application. The application associated with the icon selected by the selection instruction is executed. It accepts instructions.

[0059] When the photographing processing unit 222 receives a photographing instruction signal from the instruction reception display unit 221, The field of view (actual field of view) of the user 201 is photographed at regular intervals. The image 233 acquired by shooting is stored in the storage device 230.

[0060] The photographing by the photographing processing unit 222 is performed once when the search mode is entered. It may be configured as follows.

[0061] The image analysis unit 223 stores the image 233 acquired by the image capture processing unit 222 in the storage device 230. Then, the image 233 is analyzed, and the icon within the image 233 is extracted.

[0062] Specifically, the image analysis unit 223 first analyzes the actual objects in the image 233 that exist in the external world. The area corresponding to the object (real object) is extracted as an object on the image. Then, the real object is extracted using area information that identifies the area of ​​the extracted object. Identify the icon associated with the object.

[0063] The object is extracted by, for example, a known contour extraction (edge ​​detection). For all the extracted objects, the icon figures 2 stored in the icon DB 231 are Determine whether it matches 31b.

[0064] The determination is performed, for example, by a known matching method based only on shape. Among the objects, an object that matches the icon graphic 231b stored in the icon DB 231 is Then, the matching icon graphic 231b is determined as the object. Identify the icon as relevant to the project.

[0065] Note that the term "match" used here is not limited to a perfect match, but includes comparison results with a certain degree of similarity or higher. nothing.

[0066] The object that the image analysis unit 223 extracts from the image 233 is a street object. Marks that are displayed, posted or displayed in stores, or printed on product tags It can be a variety of real objects, such as marks that are painted or It is printed on a card or displayed on a tablet or information board. This includes various forms, such as those shown.

[0067] In addition, when there are multiple objects determined to be icons, the image analysis unit 223 These are displayed on the display 215. Then, the selection instruction is received via the instruction reception display unit 221. At this time, the display 215 displays the It may be a matched icon graphic 231b or an image data of the extracted object area. The data itself may be used.

[0068] The icon may be displayed directly on the retina of the eye. The display method is not important as long as it can display the external world scene present in a visually perceptible manner.

[0069] When a selection instruction is received for a displayed icon, the instruction reception display unit 221 Display selected icons differently from non-selected icons, e.g. by changing their size , change the color, surround it with a special frame, etc. You can also increase the size of only the selected icon. and make other icons smaller than the selected icon. Only the selected icons are displayed in color, and the other icons are displayed in grayscale, etc. This is also possible.

[0070] The application launch unit 224 executes an application associated with the selected icon. Specifically, the application launch unit 224 refers to the application DB 232 and Identify the application associated with the selected icon. Extract the application storage location 232d registered in association with the icon ID 231a / 232a. Then, the program for that application is loaded from that location into the RAM 212 and started. To make.

[0071] The application launch unit 224 launches the application of the selected icon in the storage device 23. If it is not stored as 0, an application acquisition signal is sent to the application acquisition unit 225. For example, the URL on the network 101 is stored in the application storage location 232d of the selected icon. If L is registered, or if the icon of the selected icon is In this case, the icon ID 232a is not registered. If a URL is registered in the application storage location 232d, the URL is also included. do.

[0072] When the application acquisition unit 225 receives an application acquisition signal from the application launch unit 224, The application associated with the icon ID 232a included in the acquired signal is stored in server 3. If a URL is received, the URL is accessed and the application is started. Get the application.

[0073] At this time, the application acquisition unit 225 acquires the application including the icon ID 232a. The application server 300 transmits a request to the application server 300 and receives the application. Download the application.

[0074] The application acquisition unit 225 also stores the provided application in the storage device 230. The storage location is associated with the icon ID 232a and stored in the application storage location of the application DB 232. The application launch unit 224 is then notified that the registration has been completed. The application launch unit 224 receives the notification and launches the application.

[0075] By registering the downloaded application in the application DB232, After that, you can continue to use the application.

[0076] In addition, if there is one object that is determined to be an icon by the analysis by the image analysis unit 223, In this case, the image analysis unit 223 may extract the image. If it is determined that the output object is an icon, the application launch unit 224 launches the icon. The application associated with the icon is executed as is.

[0077] In addition, although not shown in the figure, the HMD 200 of this embodiment further includes the operation of each hardware. Specifically, the communication control unit controls the operation of the communication I / F 217. an outward camera control unit for controlling the operation of the outward camera 214a; and a sensor control unit that controls the operation of each of the sensors 218.

[0078] The communication control unit is a WiFi, Bluetooth (registered trademark) and Controls wireless communication functions of mobile communication systems such as G and 5G, and transmits and receives data with external devices. The sensor control unit outputs the signals detected by each sensor.

[0079] [Processing flow] Next, the flow of application control processing by the application control unit 220 of this embodiment will be described. FIG. 5 shows a process flow of the application control process of this embodiment. The process of is started when a search instruction is received.

[0080] The photographing processing unit 222 causes the outward camera 214a to photograph and obtain an image including the actual field of view of the user 201. The image 233 is acquired (step S1101). The acquired image 233 is stored in the storage device 230. Remember.

[0081] The image analysis unit 223 analyzes the image 233 and extracts the object (step S 1102). Then, it is determined whether the extracted object is an icon or not, and the icon If so, the icon is identified (step S1103). 1, an icon graphic 231b having a shape similar to that of the object is registered. In other words, the extracted object is judged based on whether it is an icon or graphic 23 similar to the object. 1b is registered in the icon DB 231, the object is compared with the similar icon The icon 231b is determined to be a similar icon. By extracting the icon ID 231a registered in association with the icon, the icon can be identified. Furthermore, the image analysis unit 223 determines whether the icon is included in the object during the analysis of the image 233. The number of objects is counted as the number of icons.

[0082] The instruction acceptance display unit 221 determines whether the number of icons is 0 (step S110 4) If the number of icons is 0, the process ends.

[0083] On the other hand, if the number is not 0, the instruction acceptance display unit 221 determines whether the number of icons is 1 or not. (Step S1105).

[0084] If the number of icons is 1 (S1105; Yes), the process proceeds to step S1108 described later. do.

[0085] On the other hand, if the number of icons is not 1 (S1105; No), that is, if the number of icons is more than one If so, the instruction acceptance display unit 221 performs the instruction acceptance process for all the identified icons. The icon graphic 231b is displayed on the display 215 (step S1106). Then, a selection instruction is accepted (step S1107).

[0086] The application launch unit 224 refers to the application DB 232 and launches the icon I of the selected icon. The application associated with D231a / 232a is registered in the HMD200. In this example, the application storage location 232d is determined to be the application storage location 232d. If the address in the storage device 230 is stored, the application is registered. It is determined that there is.

[0087] If an application is registered in the HMD200, It is started (step S1109) and the process ends.

[0088] On the other hand, if the application is not registered in the HMD 200, the application acquisition unit 2 The application acquisition unit 225 instructs the server 300 to acquire the application. The icon ID 232a of the icon is associated with the icon ID 232a (step S1111). The application is acquired (downloaded) from the server 300 (step S11 12). The application acquisition unit 225 then downloads the application and displays it as an icon. The application ID 232a is associated with the application DB 232 and registered in the application DB 232 (step S11 BR>P3). Then, the process proceeds to step S1109.

[0089] In this embodiment, during the search mode, the application control unit 220 performs the following at predetermined time intervals: Execute the above process.

[0090] In addition, when each process transitions, the previous display is erased and a new display is made. This makes it possible to prevent unsightly images from being displayed in an unnecessary way. is applicable to all screen transitions in this embodiment.

[0091] [Usage example] Next, a specific example of use of the application control system of this embodiment is shown in FIG. 7(c) will be used to explain this. Here, when a user 201 is near a taxi stand, Let's take the example of launching a ride-hailing application. is an application for requesting, booking and fare tariffs for taxis. It is.

[0092] As shown in FIG. 1(b), when a user 201 comes near a taxi stand, for example, You will see a signboard 501 with the same shape as the icon of the ride-hailing application. When a user 201 sees a signboard 501, the user 201 looks at the signboard 501 and sees the signboard 501 displayed on the display 215. The search instruction button 711 is pressed.

[0093] In this case, the content seen by the user 201 through the display 215 is shown in FIG. As mentioned above, the display 215 is a transmissive type. In this figure, the image generated by the HMD 200 appears superimposed on the real image 510 in the real space. In the figure, the real image of the real space (real environment) is shown with a solid line, and the image generated by the instruction reception display unit 221 is shown with a broken line. In this figure, the search instruction button 711 is an image.

[0094] This place is, for example, a station square. Within the visual field of the user 201, real objects in the real space are As a project, a bus stop with a bus symbol 511 and a train platform with a train symbol 512 are shown. Among them, there is a signboard with a taxi shape 513, a person 514, a flower 515, a weed 516, and a dog 51 There is a 7th magnitude.

[0095] When it is accepted that the user 201 has selected the search instruction button 711 by line of sight, The reception display unit 221 causes the photographing processing unit 222 to acquire an image. The image field of view is preset to include the user's actual field of view through the display 215 of the HMD 200. The image 610 acquired at this time is shown in FIG.

[0096] The image analysis unit 223 extracts the object from this image 610. An example of an object is shown in FIG. 6(c). From an image 610 taken of such an environment, shape 611, bus shape 612, taxi shape 613, person shape 614, flower shape 615, weed shape 616, dog shape 617, circle 618, ellipse 619, etc. are objects. Then, the image analysis unit 223 performs the following for each of the extracted objects: Then, the icon DB 231 is referenced to determine whether or not it is an icon.

[0097] That is, the image analysis unit 223 analyzes these objects as icons in the icon DB2. 31, and whether there is a match or similar icon figure 231b registered in Then, the icon figure 231b registered in the icon DB 231 is If there is a similar object, the extracted object is identified as the icon graphic 231b. do.

[0098] In the above example, the bus symbol 611, the train symbol 612, and the taxi symbol 613 are Each icon is identified and its icon ID 231a is specified.

[0099] Then, the instruction acceptance display unit 221 displays the icon graphic 231b of each of the identified icons as follows: The icon is extracted from the icon DB 231 and generated as a display icon 712. An example of the display of the con 712 on the display 215 is shown in FIG. The display area of ​​the icon 712 is determined in advance. The display area can be changed according to an instruction from the user 201. It may be possible to change it more.

[0100] Then, when a selection instruction is received from the user 201, the icon corresponding to the Start the application you are using. In this example, the taxi icon is selected. A corresponding taxi dispatch application is launched.

[0101] At this time, the instruction acceptance display unit 221 displays the display icon 712 for which the selection instruction has been accepted, in a different manner. For example, as shown in FIG. 7(b), It is displayed larger than the other display icons 712.

[0102] After that, the instruction reception display unit 221 performs the process shown in FIG. ), the initial screen data 713 of the started application is displayed on the display. Display it on 215.

[0103] As described above, the HMD 200 of the present embodiment is configured to display the outward facing image in accordance with the instruction of the user 201. Analyze the image 233 captured by the camera 214a and extract real objects existing in the external world. , respectively, identify the relevant icon. That is, in this embodiment, the application is started by using a button similar to an icon. The application launches simply by taking a photo of a real object.

[0104] In this way, according to the present embodiment, various objects placed in the external world can be displayed as if they were real objects. It is used like an icon to start the corresponding application. If you do this, you can find the applications that are most likely to be used in the environment where the user 201 is located. It can be started without bothering the user 201. Since there is no process for accepting the selection, the user's 201 field of vision is not obstructed, and the convenience and This makes it possible to provide an HMD 200 that is easy to use and has excellent usability.

[0105] In addition, according to the present embodiment, the user 201 may be in an environment where the user 201 is likely to use the Even if multiple application icons are identified, only those are displayed That is, only icons that are likely to be used are selectively displayed.

[0106] Generally, devices based on smartphones, tablets, and PCs, etc. All icons for the applications available on the device are displayed, and you can select The user 201 must make a selection. For this purpose, a number of icons are displayed on the display 215. It will be displayed.

[0107] However, according to this embodiment, only the minimum number of icons are displayed. Therefore, even in a display terminal with a small display area such as the HMD 200, the field of view of the user 201 In addition, it is less of a hassle to search for the desired icon among many icons. Therefore, it is an HMD with high visibility, convenience, and operability. We can provide 200.

[0108] In this embodiment, the application that the user 201 wants to execute is displayed on the HMD 200. If it is not registered, the application acquisition unit 225 acquires it from the server 300 and registers it. Therefore, the probability of the desired application being executed is increased.

[0109] <Variation 1> In the above embodiment, the instruction reception display unit 221 receives an instruction from the user 201. The instructions are received by analyzing the gaze of the 201. However, the method of receiving instructions is However, the present invention is not limited to this. For example, sensing using the touch sensor 218e may be used. Within the display area of ​​the display 215, cursor movement is performed by operating the cursor display. The area is selected by a selection operation, such as a double tap operation on the touch sensor 218e. Recognize and accept what has been said.

[0110] As shown in FIG. 8(a), the instruction acceptance display unit 221 receives and displays the movement of the user's 201 fingers. is detected by the outward camera 214a. When a gesture operation is detected as a finger movement, The instruction acceptance display unit 221 may determine that the display icon 712 is selected. The gesture operation is, for example, tapping the area where the display icon 712 is displayed. These include actions such as a tap, a click, or a pinch. In addition to the finger overlapping with the display icon 712 in the line of sight, the touching action It may also be a touch action on the actual object.

[0111] <Variation 2> In the above embodiment, as shown in FIG. 7(a), the display icon 712 is However, the present invention is not limited to this. For example, the instruction reception As shown in FIG. 8(b), the additional display section 221 displays the user's 201 The display icon 712 of the virtual object is placed at the position of the original real object in the real field of view. It may be displayed as follows.

[0112] At this time, the corresponding application is stored in the storage device 230 and The display mode of the display icon 712 may be changed to other icons.

[0113] Furthermore, the icon 712 may be other than the icon figure 231b. The area of ​​the display 215 where the object is determined to be an icon may be A display that indicates that the object is an icon may be displayed. For example, a dotted line For example, a frame may be displayed at the position of the object.

[0114] <Modification 3> The display of the display icon 712 can be switched between display and non-display according to an instruction from the user 201. In the case where the display icon 712 is not displayed, the user 201 The area corresponding to the real object is selected as an icon, and the corresponding app In this case, the instruction acceptance display unit 221 may start the image capturing of the visual field range. By analyzing the image, the object in the image area selected by the user 201 is extracted. Identifies icons and accepts instructions. Also, determines when the display interferes with perception of the outside world. In this case, the instruction acceptance display unit 221 may prohibit the display of the display icon 712. In this case, the instruction reception display unit 221 acquires an image of an area corresponding to the actual field of view before displaying the image. The image analysis unit 223 analyzes the image, and the display icon 712 displays the real objects in the outside world. Also, according to the analysis result of the image analysis unit 223, it is determined whether or not the recognition of the finger is hindered. The display unit 221 is configured to display the image in a manner that does not impede the recognition of real objects in the outside world as much as possible. The display icon 712 may be adjusted to be displayed in a suitable area.

[0115] <Variation 4> In the above embodiment, the image analysis unit 223 analyzes the user 2 from the acquired image 233. The object is extracted from the actual field of view of 01. The object is not limited to a shape. For example, it may be a character string displayed on a signboard. Examples of character strings 621, 622, and 623 to be extracted as the character strings are shown in FIG. 9(a) to FIG. 9(c).

[0116] Extraction of character strings is performed using existing technology, such as OCR processing. Then, image analysis The unit 223 analyzes the meaning of the extracted character string. The semantic analysis can be performed by using existing techniques, such as morphological analysis. The analyzed meaning is compared with the contents of the icon DB 231. d and judges whether or not there is a corresponding icon. If the meaning obtained by the analysis matches the content 231d with a predetermined accuracy, the content 231 The icon having d is determined to be a related icon.

[0117] When an icon related to the extracted character string is identified, the image analysis unit 223 The icon figure 231b of the icon is extracted, and the instruction acceptance display unit 221 displays the display icon 7 12 on the display 215. The display position is determined based on the character string in the actual field of view. Examples of the display on the display 215 are shown in FIG. 9(d) to FIG. 9(f). FIG. 9(f) shows an example in which multiple icons are identified as related icons. In this case, the icon graphics 231b of all the identified icons are displayed.

[0118] By not limiting the objects to be extracted to shapes, the system can extract a wider range of objects from the user's 201 real view. The area corresponding to the icon can be extracted from the field. This increases the probability that the application will be launched. This improves convenience for the user 201. .

[0119] <Variation 5> In the above embodiment, when the number of icons is one, the corresponding application is automatically However, the present invention is not limited to this. For example, before the start of the program, the user 201 may request the program to be started. In other words, the above step S11 may be configured to receive a command for execution. Before 09, an execution instruction is received and the application launch unit 224 launches the application. .

[0120] At this time, as shown in FIG. 8(c), the instruction acceptance display unit 221 displays, for example, In a predetermined area of ​​215, an instruction button 715 for accepting an execution instruction is displayed. A selection made by guiding a line of sight to the instruction button 715 is accepted as an execution instruction.

[0121] By accepting an execution instruction from the user 201 before starting the application, unnecessary applications are eliminated. This can reduce the number of startups.

[0122] In addition, when the application is started automatically, the user 201 can start the application after the automatic start. Alternatively, the apparatus may be configured to be able to accept an instruction to stop the operation midway (an instruction to stop the operation). The stop instruction may be, for example, a gesture operation by the user 201 as described with reference to FIG. Gesture operations can be performed by waving your hand, shaking your head, etc. good.

[0123] Furthermore, the gesture operation may be configured to cancel unintended displays. For example, the user 201 may unintentionally select the search instruction button 711 due to the line of sight of the user 201. At this time, the display icon 712, the initial screen data 713, etc. that are subsequently displayed are In such a case, the instruction acceptance display unit 221 does not accept the cancellation instruction. When the user clicks the button, the application control process is stopped and the screen returns to the previous one.

[0124] In addition, if the application is started automatically without receiving an execution command, Specifically, the start prohibition of the corresponding application in the application DB 232 may be set in advance. Set a flag in the application record to prohibit launch. If the start prohibition is set, the application 224 starts the application in step S1109. For example, the application is not started, and the user 201 is notified that the application is prohibited from starting. .

[0125] <Variation 6> In the above embodiment, in the application DB 232, one icon ID 232a For each of the devices, one application is associated, but this is not limited to this. For example, Multiple applications may be associated with one icon ID 232a. stomach.

[0126] For example, the icon of a taxi (icon ID 232a is taxi 01) has multiple App D, where multiple taxi company dispatch applications are associated with each other An example of B232 is shown in Figure 10(a).

[0127] In this case, the application launch unit 224 receives the icon ID 23 of the icon for which the selection instruction has been received. 2a to output information on all applications associated with the instruction acceptance display unit 221. For example, the application name 232b is output. In response to this, the instruction acceptance display unit 221 outputs the application name 232b as shown in FIG. As shown in Fig. 0(b), image data that can identify the associated application (application A real icon 714 is generated and displayed on the display 215.

[0128] The instruction acceptance display unit 221 accepts an instruction for selection by the user 201 for the application icon 714. Then, the application launch unit 224 launches the application.

[0129] In addition, multiple applications are associated with one icon ID 232a. If so, the application is given a priority and the application icon 714 is In this case, the priority may be stored in the application DB 23 in advance. Register for 2.

[0130] When a priority order is set, the instruction acceptance display unit 221 Alternatively, it may be determined that the one with the highest priority has been selected without waiting for the selection of the other one.

[0131] In addition, if the corresponding application is not registered in the application DB 232, the application acquisition The unit 225 acquires the information from the server 300 as described above. There may be multiple applications for one icon. In this case, Before downloading, the instruction acceptance display unit 221 displays information for identifying each application. The display 215 is configured to display the image and to accept a selection instruction from the user 201. Good too.

[0132] When the application acquisition unit 225 acquires an application from the server 300, There may be no application associated with the icon ID 232a you are trying to send. In such a case, the AI ​​(Artificial Intelligence) on the server 300 The relevance function allows you to search for highly relevant (suitable) applications based on their relevance (suitability). Alternatively, a predetermined number of the highest values ​​may be selected.

[0133] The 714 app icons selected as the result are displayed in order of suitability. The display may be in order of popularity. When the instruction is received, the application acquisition unit 225 downloads the selected application to the server 30. Download from 0.

[0134] <Variation 7> In addition, the display icon 712 or the icon 713 may be displayed in cooperation with other functions of the HMD 200. The number of displayed pre-icons 714 may be reduced. This may reduce the processing load on the image analysis unit 223. At the same time, the probability of identifying the icon as the desired icon is increased. It is possible.

[0135] For example, depending on the situation of the user 201, the icon to be matched in the icon DB 231 is In this case, the image analysis unit 223 first extracts objects from the image using the above method. Then, when determining whether it is an icon or not, the type 231c that matches the situation is extracted. or icon graphics 231b of icons having content 231d.

[0136] For example, if the HMD 200 has a schedule management function, the information Good. For example, if you know that you are going to take a train soon, The image analysis unit 223 refers to the content 231d or the type 231c, and selects train-related icons. Only the icon figure 231b of the train is extracted and matched. Icons that are registered as train-related in the content 231d, or icons that are registered as train-related in the type 231c The icon is a traffic icon. The image analysis unit 223 analyzes the schedule from the schedule management function. Schedule information is obtained and the matching range is identified through semantic analysis.

[0137] In addition, for example, if the HMD 200 has a function for storing a processing history, the information can be used to The HMD 200 may store the processing history in association with the position information at the time of processing. If the system has the function to use schedule information, the system may use that information. This may be used in combination with other information.

[0138] For example, the user 201 may have a history of making payments with electronic money at a particular store. If the image is recorded as such, the image analysis unit 223 uses this information to limit the range of matching. The information used to identify the store is the store's location information. The signal is acquired by a GPS receiver 218a, which is an acquisition device.

[0139] That is, when the current position of the HMD 200 is the store, the image analysis unit 223 extracts The object is stored in the icon DB 231 as an icon graphic 231 related to electronic money. For example, information indicating that it is electronic money is registered in the content 231d. Icons that are registered as payment icons or icons that are registered as payment icons in the category 231c. do.

[0140] Furthermore, at certain stores, it is recorded that payments are made with certain electronic money. In this case, when the user 201 turns on the search mode in the store, the application launch unit 224 In this case, the application related to the specific electronic money may be directly started. The application launch unit 224 receives location information, a search instruction, and the like via the instruction reception display unit 221. Receive.

[0141] In addition to the current location information, information about the location of the current location may also be obtained. Depending on the information, the range of icons to be extracted may be limited.

[0142] For example, if the current location is a government facility, the icon for government services will be displayed with the icon for transportation. If it is a checkpoint, the search range of the image analysis unit 223 is limited to transportation icons.

[0143] This means that even if you inadvertently take a photo of a real object that resembles an icon, it will not be recognized as an icon. The current location is not extracted as a GPS receiver. Current location information obtained by 218a, a mobile communication function of the communication I / F 217, etc. This is determined using current location information, map information, etc.

[0144] In addition, it may be linked to other databases. For example, as shown in FIG. The available applications are stored in a database (store DB 235) and This may be used to reduce the number of icon graphics 231b to be matched.

[0145] The store DB 235 stores location information and available application information for each store. In the example of FIG. 11, the icon ID 232c or the icon ID 232a is registered. The image analysis unit 223 uses the GPS receiver of the HMD 200 to register the image. If the current location information acquired by the transmitter 218a matches the location information of the store, Then, the available icon ID 232a registered in the store DB 235 is This limits the range of matching.

[0146] In addition, in the case where the store DB 235 is included, when it is determined that the user 201 is in a store, When the search mode is turned on, the instruction reception display unit 221 instructs the image analysis unit 223 to analyze the image. Instead of analyzing the icon, the icon shape 2 of the icon ID 232a registered in the store DB 235 is 31b may be extracted from the icon DB 231 and displayed on the display 215. .

[0147] In addition, the store DB 235 further stores the priority of available applications for each store. In this case, the instruction reception display unit 221 may be set to a search mode in the store. When this is turned on, the application with the highest priority among the applications registered in the HMD200 will The application with the higher priority may be launched by the application launch unit 224.

[0148] In particular, when this modified example is applied to application control processing such as during payment, for example, The HMD 200 displays icons corresponding to real objects in the field of view. Even if you have a payment application, icons that are not related to payment will not be displayed. This allows users to select only the most relevant options, Improved usability.

[0149] In this modified example, when multiple icons of the same type are displayed and a selection is made according to a predetermined rule, The predetermined rule can be applied, for example, to the frequency of past use, a predetermined priority, etc. It is.

[0150] <Variation 8> In addition, an auxiliary code such as a QR code (registered trademark) may be placed near the object that corresponds to the icon. When the auxiliary code is placed, the auxiliary code is For example, the application may use auxiliary information provided by the auxiliary code. The auxiliary code may be obtained according to the information in the code. In the following, the case where the auxiliary code is a QR code is assumed. Let me explain with an example.

[0151] In this embodiment, the object extracted by the image analysis unit 223 is determined to be an icon. If so, the image analysis unit 223 further analyzes the QR code of the object in the image 233. See the code's supporting information.

[0152] If there is no QR code in the image, the image analysis unit 223 does not need to perform this process. If there are multiple QR codes in the image, for each object, select the The QR code that is closest in distance is determined to be the QR code of the object in question.

[0153] Then, the image analysis unit 223 extracts information (called QR information) from the identified QR code. The system obtains the QR code, processes it according to the QR code, and displays the results.

[0154] The image analysis unit 223 may, for example, store a location of an application related to the QR information. If the URL is registered, the URL is transmitted to the application acquisition unit 22 via the application launch unit 224. The application acquisition unit 225 accesses the URL via the communication I / F 217. Then, the application is acquired from the server 300 having the URL and is started. The QR information itself may be converted into text and displayed on the display 215.

[0155] For example, as shown in FIG. 12(a), a signboard 501 with a taxi symbol 513 is Assume that a QR code 531 is attached to the image. The image analysis unit 223 analyzes the image of the field of view. The area corresponding to the taxi symbol 513 in the image 233 is identified as an icon. 233, identify the nearest QR code, obtain its information, and use it as described above.

[0156] As shown in FIG. 12(a), a QR code placed on a signboard 501 at a taxi stand For example, the URL information of a company that provides a taxi dispatch service, a taxi dispatch application URL information for the application, location information showing the designated location for dispatch, telephone number information for the dispatch center, etc. are recorded.

[0157] In addition, the QR code placed near the icon with type 231c being shopping For example, the URL information of the online shop application, store information, etc. Information recorded includes the URL, product information at the store, and payment information at the time of checkout.

[0158] In addition, the QR code placed near the bus route image displays bus route information, bus stop information, URL information of the site that publishes fare information, etc., and URL of the site that provides real-time train information L information, URL information of the site that provides transfer information, etc. are recorded.

[0159] In addition, QR codes placed near the toilet symbols show the location of public toilets and the current The URL information of the site that provides guidance information from the current location to the nearest public toilet is recorded. Such QR codes are installed, for example, in large stores and shopping malls. .

[0160] In addition to the auxiliary information obtained by the QR code, the application launch unit 224 However, by referring to the application DB 232, the application registered in association with the icon is You may also start the following.

[0161] The auxiliary information may also be used in the processing of the launched application.

[0162] In this case, the launched application will display the auxiliary information together with the information it has read, or This allows the application to process the auxiliary information obtained from the QR code. It can automate more complex processes, such as using the data as parameters when executing an application. Therefore, by using auxiliary code in addition to the processing operations performed by the application software, This makes it possible for the user 201 of the HMD 200 to obtain more desirable processing.

[0163] For example, as shown in FIG. 12(b), a mark 518 and a QR code are attached to a tag or the like attached to clothing. The mark 518 is attached to the online clothing store. The store application is assumed to be associated with the QR code 531. The manufacturer code, product number, color, size, and other information that identifies the clothing are supplementary information. The information is registered as such.

[0164] When the app is launched, it automatically searches for the same item as the clothing by importing auxiliary information. You can order the item from the online store. You can also search for different colors and sizes of the item. You can search for it.

[0165] Also, for example, as shown in FIG. 13(a), near the cash register (register) 542 of the store Usually, there are marks for credit cards and electronic money that can be used. Marks of other available payment methods will be affixed. In such areas, receipts5 QR code 531 described in 41 or the like may be used. In this case, the QR code 531 has The auxiliary information is information necessary for payment, such as the total purchase amount, store information, purchased item information, and payment institution. The payment application that was started by selecting the icon is recorded as this. The payment process can be carried out using the auxiliary information recorded in the QR code 531. In addition, a QR code may be displayed on a display terminal such as a tablet instead of the receipt 541. In addition, there are also symbols for credit cards and electronic money that can be used. In addition, marks of other available payment methods may be displayed on the display terminal.

[0166] The photographing of the field of view and the photographing of the receipt 541 may be performed separately. The application control unit 220 extracts and identifies the user 201 from the image of the visual field range previously captured. After launching the application corresponding to the icon, take a picture of the receipt 541 and scan the QR code. 531 and reflects it in the processing of the application.

[0167] In the above description, the image analysis unit 223 also functions as a QR code reader. However, the QR code reader function can be implemented separately from the image analysis unit 223. In this case, the image analysis unit 223 may identify the QR code and then The read instruction is sent to the read function.

[0168] In the above description, the QR code is read automatically. The code may be read only when an instruction from the user 201 is received. In this case, the instruction reception display unit 221 receives an instruction from the user 201, and the image analysis unit For example, if the line of sight of the user 201 is If the orientation of the QR code on the image displayed on the display 215 is the same as the orientation of the QR code on the image displayed on the display 215, it is determined that the QR code is selected. do.

[0169] In the above example, the object and the QR code are independent. There are QR codes with icons that combine a conic shape and an auxiliary code (QR code).

[0170] In the case of such an icon-attached QR code, the image analysis unit 223 extracts the object The system does not check whether the item is an icon or not, but judges it to be an icon and records it in the QR code. The auxiliary information stored in the file is then acquired, and subsequent processing is carried out according to the auxiliary information.

[0171] FIG. 13(b) shows an example of a QR code 532 with a payment icon. In the code section, it is stated that the icon type is a payment type, and the available payment methods, etc. It will be recorded.

[0172] The image analysis unit 223 acquires this information by analyzing the image. Access the icon DB 231 and extract all icon shapes whose type 231c is a payment type. Then, as shown in FIG. 13(c), the display 215 displays the result. This is similar to the embodiment.

[0173] In this example, the icon graphics of the payment system are used as an example, but the present invention is not limited to this. It can be applied to various services and processes such as transportation, commerce, and administrative services. can.

[0174] <Variation 9> In the above embodiment and modified example, the HMD 200 itself includes the icon DB 231. The image analysis unit 223 refers to the icon DB 231, and the extracted object is Similarly, for applications, the HMD20 0 itself includes an app DB 232, and according to the app storage location 232d of the app DB 232 , and launching the application.

[0175] However, the icon DB231 and the application DB232 are provided by the HMD200 itself. Instead of the icon DB 231, the server 300 stores various icon shapes. The icon bank 311 may be provided for registering the application. It may be configured to always obtain the application from the server 300 .

[0176] In this case, as shown in FIG. 14, each icon registered in the icon bank 311 is The icon ID 311a, the icon shape 311b, the type 311c, and the content 311d are provided. Furthermore, the icon stores the URL information of the application corresponding to the icon. A corresponding application URL 311e and related information 311f are registered.

[0177] The type 311c is information classified by icon type, similar to the icon DB 231. For example, the icon ID 311a is used to uniquely identify each icon. The corresponding application URL 311e is information for identifying the application associated with the icon. The related information is, for example, information on the registration source of the icon, the rights Information on interest rates, etc.

[0178] The icon bank 311 is registered in the server 300 in which the application is registered. The server 300 may be a different device from the server 300 .

[0179] In this case, the image analysis unit 223 performs the icon identification process in step S1103. The icon bank 311 of the server 300 is accessed via the communication I / F 217, and a query is made. Then, the image analysis unit 223 calculates the icon ID 311a of the identified icon. and icon graphic 311b are downloaded to the HMD 200, and the icon graphic 311b is The icon is displayed on the display 215 and the selection is accepted. Other information (type 311c, content 311d, corresponding application URL 311 e, etc.) will also be downloaded.

[0180] The instruction acceptance display unit 221 notifies the application acquisition unit 225 of information on the selected icon. Then, the application acquisition unit 225 acquires the corresponding application U associated with the selected icon. Follow the instructions in RL311e to access the URL and download the corresponding application. And start it.

[0181] When determining whether an extracted object is an icon, only the icon DB 231 is used. If the matching range is set to , the processing speed is fast. On the other hand, by using the icon bank 311, The server 300 is accessed via the network 101, and the overall process This will slow down the processing speed, but it will increase the chances of it being identified as an icon.

[0182] In this way, by registering various icons and their related information in the icon bank 311, When you use an icon for the first time, it will recognize it and use the related application. This further improves convenience.

[0183] The application itself may also be registered in the icon bank 311 .

[0184] In addition, the above embodiment and this modification may be combined. After extracting the object, the icon DB 231 is accessed. If the icon corresponding to the extracted object is not registered in the icon DB231, The image analysis unit 223 then queries the icon bank 311 of the server 300. When icon data, such as icon figures, is downloaded from the bank 311, the icon DB 231 Add to registration.

[0185] As a result, even if the icon DB 231 is not fully populated, If we can extract objects that are similar to icons in the world, we can create a This means that applications with icons that are not widely used can be launched. For the app, prior registration will no longer be required, further improving convenience.

[0186] In addition, if the icon pattern 311b that matches the icon pattern 311b is not registered in the icon bank 311, In this case, the object extracted from the image 233 is determined to have no similar icon. The user 201 may then display a message to that effect on the display 215 and interrupt the process. may be notified accordingly.

[0187] By using the icon bank 311, various objects existing in the real world can be It can be used by registering it as an icon and assigning it a meaning or classification. Even if an object has not been tagged, the shape, location, etc. of the object may be It is also possible to infer meaning from this.

[0188] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, the display is In this embodiment, the display is connected to a mobile terminal. , on the display of a mobile device.

[0189] In this embodiment, the HMD 200 communicates with a mobile terminal of the user 201 via WiFi. The HMD 200 communicates with the user via wireless communication such as Bluetooth or Bluetooth®. Sends icon images and initial screen data of launched applications to mobile devices The mobile terminal displays the received icon image and initial screen on the display of the mobile terminal. do.

[0190] First, the system configuration of the application control system 100a of this embodiment will be described. FIG. 15(a) shows a system diagram of the application control system 100a according to the present embodiment. The application control system 100a of the present embodiment includes an HMD 200. The system includes a server 300 and a mobile terminal 400.

[0191] As in the first embodiment, the HMD 200 and the server 300 communicate with each other via the network 101. The configuration and functions of these are the same as those in the first embodiment.

[0192] However, as described above, in this embodiment, the HMD 200 and the mobile terminal 400 are in a short distance. Data is transmitted and received by wireless communication. For this reason, the communication I / F 217 of this embodiment is a LAN (Local Area Network) communication I / F and short-range wireless communication I / F Prepare.

[0193] The LAN communication I / F is an access point that is connected wirelessly via Wi-Fi (registered trademark) or other means. It is connected to the network via an AP (Access Point) device and shares data with other devices on the network. This is an interface for sending and receiving data.

[0194] The short-range wireless communication I / F transmits and receives data wirelessly with other devices equipped with wireless communication devices. For example, by using Bluetooth, information devices can be connected at distances of several meters to several tens of meters. It is an interface that uses radio waves to enable simple information exchange between devices. The I / F is, for example, a near field communication (NFC) ation)) I / F, which operates over extremely short distances of a few centimeters to about one meter, and It is also possible to realize two-way communication between devices equipped with a C chip. For example, It supports services that use contactless IC chips such as electronic money.

[0195] Furthermore, the instruction acceptance display unit 221 receives an instruction from the mobile terminal 400 via the communication I / F 217. Accept instructions.

[0196] The mobile terminal 400 is, for example, a smartphone, a tablet terminal, or the like, and has a communication function and a display device. The mobile terminal 400 may be a wearable device such as a watch. When a wearable device is used, the system can be controlled by the wearable device alone in combination with the HMD200. This system can be constructed so that the advantage of not blocking the user's hands is maintained. 00 is connected via short-range wireless communication. Short-range wireless communication is, for example, WiFi or Blu-ray. The mobile terminal 400 is further connected to the network 101. will be done.

[0197] FIG. 15B shows a hardware configuration diagram of the mobile terminal 400 of this embodiment. The HMD 400 basically has the same configuration as the HMD 200 shown in FIG. 11, RAM 412, ROM 413, and camera 414 (outward camera 414a and A line detection camera 414b), a display 415, and a voice interface (I / F) 41 6, a communication I / F 417, and a sensor 418 (GPS receiver 418a, gyro sensor 41 8b, acceleration sensor 418c, geomagnetic sensor 418d, touch sensor 418e), and each part The HMD 200 shown in FIG. 2 has the same configuration as the HMD 200 shown in FIG. It has Noh.

[0198] That is, the communication I / F 417, like the communication I / F 217, also has a LAN communication I / F and a short distance The communication I / F 417 further includes a wireless communication I / F. A telephone network communication I / F that transmits and receives calls and data via wireless communication with base stations, and a USB The HMD200 can be connected to the computer via a wired connection such as Universal Serial Bus (Universal Serial Bus). A priority communication I / F that transmits and receives data to and from other nearby devices and a proximity wireless communication device that A short-distance wireless communication interface that transmits and receives data wirelessly with devices, and an infrared communication interface that realizes infrared communication and an infrared communication I / F.

[0199] In this embodiment, the display 215 of the HMD 200 in the first embodiment The collected data is displayed on display 415 instead of display 215.

[0200] In addition, instructions from the user received via the display 415 are transmitted to the communication I / F 417. and is transmitted to the HMD 200 via the communication I / F 217. will be sent to.

[0201] Display examples on the display 415 of this embodiment are shown in FIG. 16(a) to FIG. 16(c). ) will be used to explain a specific example. For example, when the user 201 comes near a taxi stand, In the first embodiment, when the search instruction button 711 displayed on the display 215 is selected, In this embodiment, as shown in FIG. 16(a), a search instruction button is displayed on the display 415. 421 is displayed. The display is performed by CPU 411 of mobile terminal 400.

[0202] When the user 201 selects the search instruction button 421, the CPU 411 Then, the CPU 411 generates a signal indicating that the selection has been accepted. , and outputs it to the HMD 200 via the communication I / F 417. This signal is received via the terminal 17 and passed to the instruction reception / display unit 221.

[0203] The instruction reception display unit 221 instructs the image analysis unit 223 to perform the photographing process in the same manner as in the first embodiment. The image analyzing unit 223 extracts an object, Then, when the image analysis unit 223 identifies the icon, In the case where there are a plurality of icons, the icon figures are transmitted to the mobile terminal 40 via the communication I / F 217. Send to 0.

[0204] On the mobile terminal 400 side, as shown in FIG. 16(b), the CPU 411 The CPU 411 then displays the icon 422 on the display 415. When the selection is accepted, the accepted selection is sent to the HMD 200 via the communication I / F 417. Send.

[0205] On the HMD 200 side, the instruction reception display unit 221 receives the selection contents via the communication I / F 217. When the selection is received, the selection is transmitted to the application launch unit 224. In the same manner as in the first embodiment, the application DB 232 is accessed and the application Then, the application launch unit 224 determines whether the application is registered. If so, the application is started. Then, the instruction acceptance display unit 221 displays the initial screen. The plane data is transmitted to the mobile terminal 400 via the communication I / F 217 .

[0206] On the mobile terminal 400 side, as shown in FIG. 16(c), the CPU 411 The surface 423 is displayed on the display 415 .

[0207] On the other hand, if the application is not registered, the application acquisition unit 225 00, and after performing the same process as in the first embodiment, The application is started and its startup screen data is transmitted.

[0208] According to this embodiment, the system cooperates with a mobile terminal 400 held by a user 201. The user 201 can then issue operational instructions using the mobile terminal 400 that he or she owns. , and easy to use.

[0209] Various modified examples of the display mode on the display 215 are shown in the display of this embodiment. can also be applied.

[0210] For example, the display of a selected icon may be changed in a different manner from that of other non-selected icons. An icon graphic is superimposed on the acquired image. When the icon figure is displayed in a superimposed manner, the acquired image is also transmitted to the mobile terminal 400. .

[0211] Furthermore, various modifications of the first embodiment other than the display mode can also be applied to this embodiment.

[0212] <<Third embodiment>> Next, a third embodiment of the present invention will be described. In addition, a beacon is added to the system configuration of the second embodiment. This embodiment will be described taking the case where a control is added as an example.

[0213] FIG. 17(a) shows the system configuration of an application control system 100b according to this embodiment. FIG. 1 is a diagram illustrating an application control system 100b according to the present embodiment. The system includes a server 300 and a beacon 800 .

[0214] The beacon 800 outputs a unique ID that is set in advance. For example, The Bluetooth Low Energy (BEACON) signal (hereafter referred to as the beacon signal) This beacon signal is sent continuously. When an application that receives the beacon signal is started, The signal is received only at terminals that have the signal.

[0215] The beacon 800 of this embodiment is a transmitter that transmits a beacon signal. 0 outputs a beacon signal once every few seconds. This beacon signal is sent, for example, at a radius of The output should be such that it can reach a range of ten meters.

[0216] As in the first embodiment, the HMD 200 and the server 300 communicate with each other via the network 101. The configuration and functions of these are basically the same as those in the first embodiment.

[0217] However, the communication I / F 217 of the HDM 200 of this embodiment receives the beacon from the beacon 800. It also functions as a signal receiver that receives BLE signals. It is possible.

[0218] Furthermore, when the HMD 200 receives a signal from the beacon 800, the HMD 200 goes into search mode. In order to achieve this, the HMD 200 of this embodiment has a first In addition to the functions of the embodiment, a beacon receiving unit 228 that performs reception processing of a beacon signal is further provided. Prepare.

[0219] When the beacon receiving unit 228 receives a beacon signal from the beacon 800, A search instruction is sent to the display unit 221.

[0220] As described above, the beacon signal is transmitted to an application that performs processing according to the beacon signal. The beacon can only be received by devices that have the beacon reception application installed. In this embodiment, the beacon reception application program is registered in the storage device 230. Then, when the HMD 200 is started or in response to an instruction from the user 201, the CP The U211 loads the beacon reception application into the RAM212 and executes it to receive the beacon. The communication I / F 217 for receiving a beacon signal and the beacon receiving unit 228 are implemented. Once launched, the device receiving application will run in the background at all times.

[0221] In this manner, in this embodiment, the HMD 200 is in an area where a beacon signal can be received. Upon entering, the HMD 200 receives a beacon signal via the communication I / F 217. The received beacon signal is transmitted to the beacon receiving unit 228 as a search instruction, and the instruction is accepted. The image is transmitted to the display unit 221.

[0222] The instruction reception display unit 221 that receives the search instruction, like the first embodiment, 2 starts taking pictures, that is, acquiring images 233. Then, the image analysis unit 223 Each time an image 233 is captured, the image 233 is analyzed to extract objects and identify icons. Then, the application launch unit 224 launches the application of the designated icon. do.

[0223] As an example, a beacon 800 may be installed in a store, restaurant, etc., as shown in FIG. An example will be described in which the beacon 800 is placed in a transaction area. The cash register is usually located near the cash register. The card's mark or the mark of an available IC card is affixed to it.

[0224] When a user 201 approaches a cash register, the HMD 200 receives a beacon signal from a beacon 800. The instruction reception display unit 221 receives the beacon signal via the communication I / F 217. Then, the image processing unit 222 acquires an image within the field of view. 3. Extract objects and identify icons.

[0225] In the checkout area, the user 201 can view, through a transparent display 215, for example, The viewer sees a scene (real image) 550 as shown in FIG. 18(b). Specifically, the viewer sees a cash register of the store. There are stickers of various credit cards and electronic money that can be used in the store. Here, by receiving the beacon signal, the image capturing processing unit 222 performs the process shown in FIG. An image 650 as shown in FIG.

[0226] The image analysis unit 223 extracts a group of objects from this image 650. An example of the object group 620 is shown in FIG. The DB231 is referenced to identify an object that is an icon.

[0227] Then, the instruction acceptance display unit 221 displays the icon figure of the object identified as the icon. 231b is extracted from the icon DB 231 and displayed on the display 215. 19(b). In this example, the user 20 can see the image through a transparent display 215. The outside world view that comes into view of 1 will be omitted. The same applies below.

[0228] The instruction acceptance display unit 221 accepts a selection instruction from the user 201. For example, the unit 221 may display the selected icon graphic 231b in a different manner from other icon graphics. For example, as shown in FIG. 19(c), the selected icon shape may be displayed. is displayed larger than other icon shapes.

[0229] The application launch unit 224 executes the application program registered in the application DB 232 in association with the selected icon. In this example, as shown in Fig. 19(d), A screen for making payment by card, indicated by an icon, is displayed on the display 215.

[0230] If no application is registered in association with the selected icon, As in the first embodiment, the application acquisition unit 225 acquires the application from the server 300 .

[0231] In this manner, in this embodiment, the neighborhood in which the objects corresponding to the icons of the outside world are arranged Then, instead of turning on the search mode, use beacon 800. When the HMD 200 receives these signals, the HMD 200 starts the application control process.

[0232] Therefore, according to this embodiment, the same effects as those of the first embodiment are achieved. According to the embodiment, the operation of turning on the search mode is not required, and the user 201 can easily This reduces space and increases convenience.

[0233] <Modification 10> In this embodiment, the beacon signal output from the beacon 800 is basically This is a signal that indicates that an object equivalent to an icon is nearby. The beacon signal transmitted by the beacon 800 may include information other than the location information. For example, this is information that limits the processing that is performed in the vicinity.

[0234] This allows the image analysis unit 223 to limit the icon types 231c to be matched. As a result, the annoyance of having to react frequently to external marks can be reduced.

[0235] For example, in the vicinity of a place where a payment is made, only icons related to the payment are identified. In this case, when the instruction reception display unit 221 receives a beacon signal, it not only outputs a search instruction but also In addition, the beacon signal is also output to the image analysis unit 223 .

[0236] The image analysis unit 223 analyzes the beacon signal and identifies the type 231c of the required icon. Then, when identifying an icon corresponding to the extracted object, the identified type Matches only icons that match 231c.

[0237] For example, the beacon signal is registered so that only icons with the type 231c being payment are extracted. In the example of FIG. 19(b), from the left, the icons to be registered in the icon DB are The icon types are "Payment, Payment, Payment, Payment, Points Granting." In this case, the icon on the far right No icon is displayed on the display 215.

[0238] In this embodiment, as in the second embodiment, the display destination is the mobile terminal 400. It's fine.

[0239] <Modification 11> The HMD 200 of each of the above embodiments and modifications further includes a user authentication function. Good too.

[0240] The user authentication function is to determine whether the user 201 of the HMD 200 is a user 201 registered in advance. Only if the authentication is successful, the application control process is instructed to be performed. This is a security feature that indicates

[0241] A functional block diagram of the HMD 200 of this modified example is shown in FIG. 20 is a function of the first embodiment, and further includes a function for authenticating a user 201 of the HMD 200. The device includes an authentication unit 229 that performs user authentication.

[0242] For user authentication, for example, iris authentication can be used. In this case, the user authentication device This function is realized by, for example, authentication unit 229 and gaze detection camera 214b.

[0243] In the iris authentication, the iris of the user 201 is photographed in advance, and the verification data 236 is stored in the ROM 213. When the user 201 wears the HMD 200, the authentication unit 229 performs line of sight detection. The camera 214b captures an image of the iris of the user 201 and compares it with a previously stored image of the iris. If the verification is successful, the authentication unit 229 determines that the user is a legitimate user 201. The instruction reception display unit 221 receives the notification of the successful matching. In this case, a search instruction button is displayed on the display 215.

[0244] Note that the user authentication is not limited to the above method. For example, the authentication pattern can be tracked by eye gaze. unlocking the device, blinking a certain number of times in a certain time, blinking at a certain time interval, or by finger It may be a fingerprint authentication function, etc. Also, it may be a combination of these. The authentication pattern may be, for example, displayed on the instruction acceptance display unit 221. Create more.

[0245] This makes it possible to prevent transactions such as payment processing from being spoofed by others. This allows us to realize the HMD200, which is convenient, easy to use, and safe. can.

[0246] <Modification 12> In the first embodiment, the third embodiment, and each modification, the display terminal is an HMD. 200 is used as an example, but the display terminal is not limited to this. In this example, a mobile device such as a smartphone or tablet equipped with a camera or communication function It's fine.

[0247] In either case, you can activate the app by simply taking a picture of a real object that resembles the icon shape. Therefore, you can launch the icon of the application you want to use. This eliminates the need to search for and launch the application on the display screen, improving usability. do.

[0248] In the first and second embodiments, when the search mode is turned on, An image 233 of the outside world is acquired and image analysis is started, but this is not limiting. For example, The image 233 of the outside world is constantly acquired at a predetermined time interval, and each time an image is acquired, the image analysis unit 2 23 may be configured to specify an icon.

[0249] In this case, the display 215 always displays the display icon 71 that can be selected at that time. When the user 201 wants to execute the application, the display A selection can be made from among the icons 712.

[0250] The present invention is not limited to the above-described embodiment and modifications, and various modifications are possible. For example, the above-described embodiment and modifications are included to clearly explain the present invention. The present invention is described in detail for the purpose of convenience, and is not necessarily limited to the embodiment having all of the configurations described. In addition, a part of the configuration of one embodiment or modified example is not intended to be used in conjunction with another embodiment or modified example. In addition, the configuration of one embodiment or modification may be replaced with another embodiment. It is also possible to add the configuration of the embodiment or modification. It is possible to add, delete, or replace part of a configuration with other configurations.

[0251] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in whole or in part by, for example, The above configurations and functions may be realized by hardware, such as by designing an integrated circuit. The processor interprets and executes the programs that realize the respective functions. It may be realized by software. Each function may be realized by a program, a table, a file, etc. The information is stored in the memory, hard disk, and SSD (Solid State Drive). It can be placed on a recording device such as a digital camera, or on a recording medium such as an IC card, SD card, or DVD. do.

[0252] In addition, control lines and information lines are shown as necessary for explanation, and may not necessarily correspond to all This does not necessarily show all control and information lines. In reality, almost all components are interconnected. It can be said that this is the case. [Explanation of symbols]

[0253] 100: application control system, 100a: application control system, 100b: application control system, 101: network, 200: HMD, 201: User, 211: CPU, 212: RAM, 213: ROM 214: camera, 214a: outward facing camera, 214b: gaze detection camera, 215: display Play, 216: Audio I / F, 217: Communication I / F, 218: Sensor, 218a: GPS Receiver, 218b: gyro sensor, 218c: acceleration sensor, 218d: geomagnetic sensor 218e: touch sensor, 219: bus, 220: application control unit, 221: instruction reception table display unit, 222: photography processing unit, 223: image analysis unit, 224: application launch unit, 225: application acquisition unit, 228: beacon receiving unit, 229: authentication unit, 230: storage device, 231: Icon DB, 231a: Icon ID, 231b: Icon shape, 231c :Type, 231d:Content, 232:Application DB, 232a:Icon ID, 232b:A Pre-name, 232c: application ID, 232d: application storage location, 233: image, 234: display data, 235: store DB, 236: matching data, 241 : display control unit, 242: display image generating unit, 243: line of sight detecting unit, 244: analysis control unit, 300: Server, 311: Icon Bank, 311a: Icon ID, 311b: Icon 311c:Type, 311d:Content, 311e:Compatible application URL, 311f: Related information, 400: mobile terminal, 411: CPU, 412: RAM, 413: ROM, 414: camera ra, 414a: outward facing camera, 414b: gaze detection camera, 415: display, 417 : communication I / F, 418: sensor, 418a: GPS receiver, 418b: gyro sensor, 418c: acceleration sensor, 418d: geomagnetic sensor, 418e: touch sensor, 419: Bus, 421: search instruction button, 422: icon graphic, 423: initial screen, 501: Signboard, 510: Real image, 511: Bus image, 512: Train image, 513: 514: People, 515: Flowers, 516: Weeds, 517: Dogs, 518: Marks , 531: QR code, 532: QR code with icon, 541: receipt, 550: image, 601: Image, 602: Object, 603: Taxi dispatch application, 610: Image, 611: figure, 612: figure, 613: figure, 614: figure, 615: figure, 616: figure Shape, 617: Figure, 618: Circle, 619: Ellipse, 620: Object group, 621: Character Column, 622: String, 623: String, 650: Image, 711: Search instruction button, 712: Display icon, 713: Initial screen data, 714: App icon, 715: instruction button, 800: beacon

Claims

1. A photographing device, which analyzes images taken by the photographing device and executes applications A display terminal including an application control unit, At the instruction of the user of the display terminal, The imaging device captures an image of the user's visual field; The application control unit controls executable applications displayed in the image of the visual field range. An icon graphic corresponding to the application is searched for based on the image in the visual field range, and the searched icon graphic is When one icon is found, the application corresponding to the icon is If multiple applications are found, the icon graphics are displayed on the display terminal. , launching an application corresponding to the icon graphic selected by the user. thing A display terminal comprising:

2. A display terminal according to claim 1, The application control unit searches for the icon graphic and the icon at a predetermined time interval. Repeat the process of launching the application that corresponds to the shape of the cone, After the process of starting the application, the icon graphic is displayed on the display terminal. If so, erase the icon. A display terminal characterized by:

3. 2. The display terminal according to claim 1, The display terminal is a head-mounted display. A display terminal characterized by:

4. An application including a display terminal and a server connected to the display terminal via a network.

1. A communication control system comprising: A photographing device; A storage device; In response to an instruction from a user of the display terminal, the image capturing device captures an image of the user's field of view. and acquiring an icon corresponding to an executable application displayed in the image of the field of view. an image analysis unit that searches for a con figure based on an image in the field of view; The application associated with the icon graphic found by the search is a start-up unit for determining whether the program is stored in the storage device; If it is determined that the application is not stored in the storage device, and an acquisition unit that acquires a corresponding application from the server, The launch unit further executes, when the number of the icon figures found by the search is one, The application associated with the icon is automatically started, and if there are multiple applications, and launching an application associated with the icon graphic selected by the user.

1. An application control system comprising:

5. 5. The application control system of claim 4, The application corresponding to the icon graphic found by the search is If the relevant application is not stored in the Select a certain number in order 1. An application control system comprising:

6. An application control method in a display terminal having an imaging device, comprising: According to an instruction from a user of the display terminal, Acquire an image of the user's field of view; Icons corresponding to executable applications displayed in the image of the field of view Searching for a shape based on the image of the field of view; If one icon is found by the search, the icon is associated with the icon. The application is automatically started, and if there are multiple applications, the icon is displayed on the display terminal. and displaying the application associated with the icon graphic selected by the user. Launch the application 23. An application control method comprising:

Images