Glasses-type information device, method and program thereof

The integration of remote control and mirroring functions between smart glasses and smart watches addresses the usability challenge, enhancing user experience and design flexibility while reducing the need for complex operation units on smart glasses.

JP7581075B2Active Publication Date: 2024-11-12CANON KK
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021024691
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-02-18
Publication Date
2024-11-12
Estimated Expiration
2041-02-18

AI Technical Summary

Technical Problem

The challenge lies in enhancing the usability of glasses-type information devices like smart glasses by enabling seamless remote control and interaction with other wearable terminals, such as smart watches, to improve functionality and user experience.

Method used

The glasses-type information device incorporates a control mechanism that allows remote operation from a cooperating device, such as a smart watch, with features like mirroring displays and remote control functions, utilizing a common operation unit and application management units to handle user inputs and environmental changes.

Benefits of technology

This solution enhances usability by allowing seamless interaction and control between smart glasses and smart watches, improving user experience and reducing the need for high-performance operation units on smart glasses, thereby contributing to design flexibility and weight reduction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581075000005
    Figure 0007581075000005
  • Figure 0007581075000006
    Figure 0007581075000006
  • Figure 0007581075000007
    Figure 0007581075000007
Patent Text Reader

Abstract

To solve the problem in which: a glasses type information apparatus such as smart glasses is likely to be worn and used daily in place of a smartphone, and in the future, it is considered that usability needs to be improved when the functionality of the glasses type information apparatus is remotely controlled from another wearable terminal.SOLUTION: A glasses type information apparatus is worn by a user carrying a cooperation device, and has: first control means that controls display such that an image corresponding to an operation unit related to remote control of an application executed in the glasses type information apparatus, the remote control performed by the cooperation device, is projected in the user's field of view; and second control means that controls the functions of the application based on information for the application received in response to the user's operations performed on the cooperation device.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to technology relating to glasses-type information devices such as smart glasses. [Background technology]

[0002] In recent years, wearable devices have become more diverse (smart glasses, smart watches, smartphones, etc.), and there are many cases where people wear multiple devices at the same time, so technologies for linking these wearable devices have been proposed. Among these, a technology has been proposed for linking a smartphone with a head-mounted display (HMD), in which partial image data, which is a partial area of ​​an image being displayed on the smartphone, is displayed on the HMD (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-36914 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the future, smart glasses and other eyeglass-type information devices may be worn and used on a daily basis, replacing smartphones. On the other hand, wearable devices other than eyeglass-type information devices, such as smart watches, are expected to be used in conjunction with them. In the future, it will be necessary to link these devices and improve the usability when remotely controlling the functionality of eyeglass-type information devices from other wearable devices. [Means for solving the problem]

[0005] Therefore, the glasses-type information device of the present invention is a glasses-type information device worn by a user having a linked device, and includes a first control means for controlling a display so that an image corresponding to an operation unit related to remote control by the linked device for an application executed on the glasses-type information device is projected into the field of view of the user, and a second control means for controlling a function of the application based on information for the application received in response to an operation of the user performed on the linked device; one or more applications can be executed on the glasses-type information device, and when an application for which a setting for displaying an image corresponding to the operation unit is enabled is executed on the glasses-type information device, the first control means projects an image corresponding to an operation unit related to a remote control of the application into the user's field of vision via the glasses-type information device, and when an application for which a setting for displaying an image corresponding to the operation unit is disabled is executed on the glasses-type information device, the first control means does not project an image corresponding to the operation unit related to a remote control of the application via the glasses-type information device. It is characterized by: Effect of the Invention

[0006] According to the present invention, it is possible to improve usability when remotely controlling an eyeglass-type information device by linking it with another wearable terminal (such as a smart watch). [Brief description of the drawings]

[0007] [Figure 1] A diagram showing an example of a system configuration including glasses-type information devices and other wearable devices. [Diagram 2] A diagram showing an example of the hardware configuration of each device [Diagram 3] A diagram showing an example of the software module configuration of each device [Figure 4] 1 is a flowchart of a process performed by a wearable terminal that cooperates with a glasses-type information device according to a first embodiment. [Diagram 5] 1 is a flowchart showing the processing of a glasses-type information device that cooperates with another wearable device in the first embodiment. [Figure 6] Example of display control on each device when remote control starts [Figure 7] Example of display control on each device when selecting a control application for remote control [Figure 8] Example 1 of display control on each device when mirroring display is performed in remote control [Figure 9]Example 2 of display control on each device when mirroring display is performed in remote control [Figure 10] Example of display control on each device when mirroring display is not performed in remote control [Figure 11] 11 is a flowchart of a process for detecting an environmental change of a glasses-type information device that cooperates with a wearable terminal according to a second embodiment. [Figure 12] 11 is a flowchart of a process performed when an environmental change is detected by a wearable terminal that cooperates with a glasses-type information device according to a second embodiment. [Figure 13] 11 is a flowchart for explaining processing on the side of a glasses-type information device that cooperates with a wearable terminal in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] <Embodiment 1> FIG. 1 shows an example of a system configuration diagram including a glasses-type information device and a wearable terminal that cooperates with the glasses-type information device. The glasses-type information device is a device that can communicate with a mobile network (mobile communication network) 130 and the Internet 140 directly or via a mobile router (not shown), and the present invention gives smart glasses 101 as an example. As glasses-type information devices, not only compound-eye glasses but also monocular smart glasses and head-mounted displays can be applied. In addition, the wearable terminal is an information device that has notification functions such as a display and vibration function, and a communication function such as Bluetooth (registered trademark) that can communicate with the glasses-type information device, and the present invention gives a smart watch 110 as an example.

[0010] The smart glasses 101 are wearable information terminals worn by a user near the eyes, and display a virtual image on a display unit 102 within the user's field of vision without blocking the user's field of vision. Such a display method is called AR (Augmented Reality) or MR (Mixed Reality), and is provided by a function of projecting information onto a transparent display (lens 105) or the user's retina (not shown). Although the figure shows a display that provides an image to one eye, the present invention can also be applied to smart glasses 101 that project an image onto the field of vision of both eyes.

[0011] The display unit 102 is also provided with a camera unit 103 for capturing an image of a subject in the user's line of sight. 104 is a frame and also has a touch sensor, and is an operation frame unit for operating the terminal. The frame 104 has a built-in speaker function and can transmit sound to the user.

[0012] The smart glasses 101 can be equipped with a built-in module such as an eSIM (embedded subscriber identity module) and can connect to the Internet 140 via a mobile network 130 that uses 4G or 5G lines. In addition, the smart glasses 101 can be connected to a mobile router owned by a user via Wi-Fi or USB, and can connect to the mobile network 130 via the mobile router. The smart glasses 101 can also connect to the Internet 140 via Wi-Fi, etc., without going through the mobile network 130.

[0013] The smart watch 110 is a wristwatch-type information terminal worn by a user on the wrist, and the display unit 111 not only displays information such as the time, but also functions as a touch panel and allows the user to operate the terminal. 120 indicates wireless communication for data exchange between the smart glasses 101 and the smart watch 110, and assumes wireless communication based on the Bluetooth (registered trademark) standard, but is not limited to this. In addition to display, the smart watch 110 also has a notification function using a vibration function.

[0014] FIG. 2 is a diagram showing an example of the hardware configuration of each device according to the present invention.

[0015] FIG. 2A is a diagram showing the hardware configuration of the smart glasses 101.

[0016] The CPU 201 collectively controls various functions of the smart glasses 101 via the internal bus 206 by a program stored in the ROM 203. The execution result of the program by the CPU 201 can be projected and displayed as an image in the user's field of vision by the display 202. In this embodiment, the display method is assumed to be a method in which the user sees what the display 202 projects in the field of vision through the transparent lens 105. However, it is also possible to adopt a method in which the display 202 projects directly onto the retina. The ROM 203 is a flash memory or the like, and various setting information and application programs, as described above, are stored. The RAM 204 functions as a memory and a work area for the CPU 201. Note that the processing shown in the flowchart that is a feature of this embodiment, which will be described later using figures, is processing that is realized by the CPU 201 executing the corresponding program.

[0017] The network I / F (interface) 205 is a network module for connecting to the mobile network 130 or Wi-Fi. This module enables connection to the Internet. When using a mobile router, it is also possible to connect using a USB I / F (not shown) of the smart glasses 101.

[0018] The operation unit 207 accepts input from the user through the operation frame 104, and transmits a signal corresponding to the input to each of the above-mentioned processing units through the operation unit I / F 208. The sensor unit 209 is a simplified illustration of one or more sensors. Specifically, at least one of a GPS, a gyro sensor, an acceleration sensor, a proximity sensor, a blood pressure / heart rate measurement sensor, and the like is implemented in the smart glasses 101. In addition, a sensor for detecting biometric information for realizing fingerprint authentication, vein authentication, iris authentication, and the like may also be implemented in the smart glasses 101. The camera 210 has a photographing function, and the photographed image data is stored in the ROM 203. The laser 211 projects various contents onto the display 202, and in the case of a retinal projection method, projects the contents directly onto the retina. The storage device 212 is a storage medium and is a device for storing various data such as applications. It also includes a device for reading data from the storage medium and a device for deleting data. Some terminals do not have the storage device 212 and only have the ROM 203. The short-range communication I / F 213 is an interface used for communication with the smart watch 110 and the like, and realizes wireless communication 120, for example.

[0019] Although not shown, the smart glasses 101 can further include a configuration for realizing voice calls using a network or telephone line so that they can be used in place of modern smartphones. Specifically, the smart glasses 101 will be equipped with a component for connecting to a telephone line, a speaker, a microphone, a voice control chip, and the like.

[0020] FIG. 2B is a diagram showing the hardware configuration of the smart watch 110.

[0021] The CPU 221 controls various functions of the smart watch 110 via the internal bus 226 in accordance with a program stored in the ROM 223. Results of application programs executed by the CPU 221, notifications from the smart glasses 101, and the like are displayed on the display 222. The ROM 223 is a flash memory or the like, and stores various setting information and application programs as described above. The RAM 224 functions as a memory and a work area for the CPU 221. Note that the processing shown in the flowchart that characterizes this embodiment and will be described later with reference to the drawings is processing that is realized by the CPU 221 executing the corresponding program.

[0022] The short-range communication I / F 225 controls the smart watch 110 to exchange data with an external network device one-way or two-way via the wireless communication 120. The operation unit 227 accepts an input from a user on the display unit 111 and transmits a signal corresponding to the input to each of the above-mentioned processing units via the operation unit I / F 228. A form in which the display 222 and the operation unit 227 are integrated using a touch panel or the like can also be realized. The sensor unit 229 is a simplified illustration of one or more sensors. Specifically, the smart watch 110 is equipped with at least one of a GPS, a gyro sensor, an acceleration sensor, a proximity sensor, a blood pressure / heart rate measurement sensor, and the like. In addition, the smart glasses 101 may be equipped with a sensor for detecting biometric information for implementing fingerprint authentication, vein authentication, iris authentication, and the like.

[0023] As mentioned above, the smart glasses 101 can further execute software for implementing a voice communication function.

[0024] FIG. 3 is a diagram showing an example of a modular software configuration of each device according to the present invention.

[0025] FIG. 3A shows the software module configuration of the smart glasses 101.

[0026] The communication unit 301 transmits instruction information to the smart watch 110 and status information of the smart glasses 101 to the smart watch 110. In addition to receiving instruction information transmitted from the smart watch 110, the communication unit 301 also receives notifications from an application server that cooperates with the smart glasses 101 via a network using the network I / F 205. The status information means whether the smart glasses 101 are being used by a user, and mode information of the smart glasses 101 (standby mode, power saving mode, etc.). The storage unit 302 exchanges information between the ROM 203, the RAM 204, and other processing units. The storage unit 302 also stores image data captured by the camera 210.

[0027] The display unit 303 displays the results of various applications executed by the CPU 201, such as AR / MR display. For example, AR technology is used to display content (information projection) on the display 202 so that virtual content is superimposed on real space. Note that the present invention can also employ a method of direct projection onto the retina. Here, in the information projection, the position and orientation of the user's own device in real space may be estimated based on a technology such as SLAM (simultaneous localization and mapping). The estimation result is used for display processing of the virtual content.

[0028] The operation unit 304 accepts input from the user through the operation frame 104, and accepts and controls various application operations and operation instructions for display on the display 202. The camera control unit 305 controls the start, stop, and shooting of the camera 210, and stores the captured image data in the ROM 203 in cooperation with the storage unit 302. The mode control unit 306 controls mode switching between multiple modes provided in the smart glasses 101, and switches the state of the smart glasses 101 to a mode desired by the user. The status detection unit 307 manages information such as status changes and usage status detected by the sensor unit 209, and stores it in the ROM 203. The pairing unit 308 controls the short-range communication I / F 213 to enable bidirectional data communication with an external network device via wireless communication 120.

[0029] The application management unit 309 manages the execution status of various applications, version information, notifications received from the server, and whether or not the power saving mode is in effect, and manages these in the ROM 203 and RAM 204 via the storage unit 302. The power saving mode (including the sleep mode) is a mode in which power consumption is controlled to be minimized when the remaining battery level of the smart glasses 110 is below a predetermined value.

[0030] In addition, the application management unit 309 controls various applications based on "user input via the operation unit 304" and "instructions received from a linked (paired) external terminal via the communication unit 301." In this embodiment, the linked external terminal is a wearable terminal worn by a user wearing the smart glasses 101, and a specific example of this is the smart watch 110.

[0031] Here, the application management unit 309 may first accept either "a user input instruction via the operation unit 304" or "an instruction from an external terminal via the communication unit 301" and then accept the other instruction. In this case, the instruction accepted first may be given priority, or the application management unit 309 may predetermine which instruction is to be given priority uniformly or for each application, or the user may set it.

[0032] Furthermore, when content is to be displayed on the display 202 during application control, the application management unit 309 displays the content on the display 202 via the display unit 303 .

[0033] The above FIG. 3(A) shows the software module configuration of the smart glasses 101 in this embodiment. However, the smart glasses 101 may be provided with a separate common operation module that can receive operation instructions independently of the various linked terminals, rather than from each of the linked terminals. The operation instructions include operation instructions for applications on the smart glasses 101 by remote control, and instructions for linking functions such as displaying / hiding mirroring display. The definition of mirroring display will be described later.

[0034] Then, common standardized instruction information may be prepared between the common operation unit and each interconnection terminal, and the instruction information may be exchanged between each interconnection terminal and the smart glasses 101. Also, the terminal common operation unit may generate or pre-store a common operation instruction UI for each interconnection terminal, distribute it to each interconnection terminal, and notify the smart glasses 101 of the received information so that each interconnection terminal displays the distributed UI and accepts user operations.

[0035] FIG. 3B shows the modular configuration of software in the smart watch 110.

[0036] The clock unit 321 manages and controls time information to be displayed on the display 222. The management and control of the time information may be performed by an application program stored in the ROM 223. The communication unit 322 transmits instruction information to the smart glasses 101, and receives status information of the smart glasses 101, as well as notifications from an application server. The storage unit 323 exchanges information between the ROM 223, the RAM 224, and other processing units. The display unit 324 displays the time information managed by the clock unit 321 and the results of various applications executed by the CPU 221. It also displays application notifications transferred from the smart glasses 101. The operation unit 325 accepts input from the user on the display unit 111, and controls the operation of various applications and display on the display 222. The status detection unit 326 manages the status changes detected by the sensor unit 229 as information and stores it in the ROM 223. A pairing unit 327 controls the short-range communication I / F 225 so as to enable two-way data communication with an external network device via the wireless communication 120 .

[0037] The application management unit 328 manages the execution status of various applications, version information, notifications received from the server, and whether or not the smart watch 110 is in sleep mode, and manages these in the ROM 223 and the RAM 224 via the storage unit 323. The sleep mode is a mode to which the smart watch 110 enters when there is no operation by the operation unit 325 for a predetermined period of time or more, and the smart watch 110 turns off the screen of the display 222, locks the screen, and so on.

[0038] Furthermore, the application management unit 328 controls various applications based on user input via the operation unit 324. When content is to be displayed on the display 222 while controlling an application by the application management unit 328, the application management unit 328 displays the content on the display 222 via the display unit 323.

[0039] Furthermore, the application management unit 328 also manages, in the ROM 223 and the RAM 224, glass linkage information for each application that determines which functions are enabled when various applications on the smart watch 110 link with the smart glasses 101.

[0040] The cooperation here assumes remote control of an application on the smart glasses 101 by the smart watch 110. During remote control, a simplified display of an operation panel corresponding to the smart watch 110 is provided as part of a projected image into the user's field of vision provided by the smart glasses 101. In this embodiment, the simplified display of the operation panel corresponding to the wearable terminal used for remote control such as the smart watch 110 is also called a mirroring display.

[0041] The per-application glass linkage information specifies, for each application, whether to enable the remote control function (ON / OFF) and whether to enable the mirroring display function (ON / OFF) (see Table 1).

[0042] [Table 1]

[0043] The ON / OFF setting value indicating whether or not to perform this mirroring display may be included in the installation information of various applications on the smart watch 101, or may be user-settable in a setting app (not shown) managed by the application management unit 328.

[0044] The above-described software module configuration of the smart glasses 101 and smart watch 110 in Fig. 3 is assumed, and the linking function between the smart glasses 101 and the smart watch 110 that is a feature of this patent is described below. The description is also premised on the operation of an application in which both remote control and mirroring display are ON in the above-described glass linking information for each application (Table 1).

[0045] The smart glasses 101 of this embodiment are equipped with a remote control function that receives user instruction information for controlling an application of the smart glasses 101 from a linked device (the smart watch 110 in this embodiment) and processes the application to provide a function based on the user instruction information. The smart glasses 101 further accept user input via the operation unit 325 on the smart watch 110 as user instruction information in order to directly control a control target application on the smart glasses 101. In addition, the user instruction information may also include an application ID on the smart glasses 101 indicating the control target.

[0046] The smart glasses 101 are assumed to be equipped with an OS function and / or a dedicated control application for issuing control instructions to the controlled application of the smart glasses 101. With this equipped functionality, data corresponding to the input user instruction information is handed over to the controlled application, and control of the target application is realized according to the user instruction. For example, the control application is an application corresponding to each application of the smart glasses 101, or an application specialized for a dedicated function such as character input or screen scrolling and commonly used. In addition, for example, when the control application is terminated, the user instruction information notified to the smart glasses 101 may include information indicating the end of control for the target application. When the application management unit 328 detects that a predetermined time has elapsed since the state of the smart watch 110 transitioned to the sleep mode, the user instruction information may include information indicating the end of control for the target application.

[0047] The smart glasses 101 are equipped with a mirroring display function that receives UI information of a linked device (smart watch 110) operated during this remote control, and projects an AR operation unit based on the UI information on the display 202. The remote control function and mirroring display function installed in the smart glasses 101 may be installed as management functions of the application management unit 309, or may be installed as one of various applications managed by the application management unit 309.

[0048] At this time, while the content of the application to be controlled by the remote control function is projected and displayed on the smart glasses 110, an AR operation unit (mirroring display) based on the above-mentioned UI information for controlling the application to be controlled by the mirroring display function is provided to the user's field of vision.

[0049] In the smart glasses 101 in this embodiment, when providing a mirroring display, the content image of the controlled application is slightly reduced to create a free space, and an automatic control is realized to provide an AR operation unit based on UI information in the free space. However, depending on the situation, the mirroring display can also be realized by displaying a part or all of the image corresponding to the AR operation unit superimposed on the content image of the controlled application being displayed. For example, when the controlled application is running in the background, the mirroring display may be displayed in full screen.

[0050] The UI information used for the mirroring display includes screen information used for projecting the operation unit of the smart watch 110. The screen information may be a copy of the appearance of the linked device (smart watch 110). In addition, this screen information may be data itself such as still images and videos, or may be realized by structured data in JSON or XML format that enables the smart glasses 101 to generate image information.

[0051] In addition, the linked device (smart watch 110) may be equipped with hardware (buttons or a mouse ball) that can be used for remote control (not shown), in which case data input by the hardware is passed as user instruction information to the smart glasses 101. In that case, the UI information includes drawing data that corresponds to the hardware.

[0052] In addition, the screen information on the smart watch 110 in the mirroring display during remote control includes screen information of control screens such as a "character input screen" and a "scroll screen" that are control information input to an application on the smart glasses 101 to be controlled. When an application during mirroring display is terminated, information indicating the end of mirroring display is included in the UI information notified to the smart glasses 101. In addition, when the application management unit 328 detects that a predetermined time has elapsed since the smart watch 110 entered sleep mode, information indicating the end of mirroring display may be included in the UI information.

[0053] Furthermore, the user can specify whether to permit / prohibit remote control and / or mirroring display from the linked device (smart watch 110) to the smart glasses 101. It is possible to specify by selecting the paired device. In addition, the user can arbitrarily specify various conditions such as an ID for identifying the linked device, a product type, an OS / OS version installed in the product, an application type / application version, etc., and the smart glasses 101 can automatically determine whether to permit / prohibit remote control and / or mirroring display for the linked device that satisfies the conditions. In this embodiment, a case will be described in which the smart glasses 101 uniformly permits remote control and mirroring display regardless of the information of the smart watch 110.

[0054] 4 and 5 show a flowchart characteristic of this embodiment when the smart watch 110 controls an application installed in the smart glasses 110 in cooperation with the paired smart watch 110 and smart glasses 101. In Fig. 4 and Fig. 5, it is assumed that the smart watch 110 and the smart glasses 101 have already been paired by the pairing unit 327 and the pairing unit 308, respectively.

[0055] 4 is a flowchart for explaining the processing in the smart watch 110. This flowchart explains the processing such as transmitting information for remote control or mirroring display to the smart glasses 101 when the user operates an application on the smart watch 110. The application management unit 328 is the subject of the processing steps included in this flowchart.

[0056] The application management unit 328 detects operation information for an arbitrary application on the smart watch 110 by notification of "user input from the operation unit 325" or "instruction information by the smart glasses 101 from the communication unit 301" (S401). At this time, the detected operation information is information such as the start of operation of the application, operation in progress, and operation end. Then, the application management unit 328 starts control of the application for which the control instruction was given, and checks the default value of remote control ON / OFF of the corresponding application in the glass linkage information for each application (Table 1) (S402). If the remote control is ON, the default value of mirroring display ON / OFF of the corresponding application is further checked in the glass linkage information for each application (Table 1) (S403). If mirroring display is ON in S403, user instruction information and UI information are transmitted to the smart glasses 101 via the communication unit 321 (S404).

[0057] If the mirroring display is OFF in S403, only the user instruction information is transmitted to the smart glasses 110 via the communication unit 321 (S405).

[0058] Similarly, when the remote control is OFF in S402, the default ON / OFF value of the mirroring display of the corresponding application is confirmed in the glass linkage information for each application (Table 1) (S406). When the mirroring display is ON in S406, only the UI information is transmitted to the smart glasses 110 via the communication unit 321 (S407). Then, when the mirroring display is OFF in S406, the flow of FIG. 4 ends.

[0059] On the other hand, if operation information for the arbitrary application is not detected in S401, the application management unit 328 determines whether the smart watch 110 is in a sleep mode (S408). Note that the determination of whether the smart watch 110 is in a sleep mode in the following determination may be replaced with a determination of whether the smart glasses 101 are in a desired power saving mode to which the state can be changed. If it is determined in S408 that the smart watch 110 is not in a sleep mode, the flow of FIG. 4 ends.

[0060] If it is determined in S408 that the mode is the sleep mode, it is further determined whether or not a predetermined time has elapsed since the transition to the sleep mode (S409). If it is determined in S409 that the predetermined time has not elapsed, the flow of FIG. 4 ends, and if it is determined in S409 that the predetermined time has elapsed, the flow proceeds to S410. In S410, information indicating the end of control by remote control is set in the user instruction information, and information indicating the end of display by mirroring display is set in the UI information. Then, in S411, the user instruction information and the UI information are transmitted to the smart glasses 101.

[0061] 5 is a flowchart of the smart glasses 101, which receives control information for remote control or mirroring display from the smart watch 110 and performs control based on the control information. The application management unit 309 is responsible for the processing steps included in this flowchart.

[0062] When the application management unit 309 receives data from a linked device via the communication unit 301, it checks whether the received data includes user instruction information (S421). If the received data includes user instruction information, it is further checked whether the received data includes UI information (S422). Then, if the received data includes UI information, starting, continuing, and ending remote control that controls an application on the smart glasses 101 is executed based on the user instruction information, and starting, continuing, and ending mirroring display is also executed based on the UI information.

[0063] If the received data does not include UI information in S422, the application management unit 309 checks whether mirroring display is in progress based on the execution status of the managed application (S424). If mirroring display is in progress in S424, it is determined whether the smart glasses 101 are in a sleep mode (S425). Note that the determination of whether the smart glasses 101 are in a sleep mode in the following determination may be replaced with a determination of whether the smart glasses 101 are in a desired power saving mode to which the state can be transitioned.

[0064] If the device is not in sleep mode in S425, remote control is started, continued, and ended based on user instruction information, and mirroring display is continued (S426). If the device is in sleep mode in S425, it is further determined whether a predetermined time has passed since UI information was last received from the linked device (smart watch 110) (S427). If the predetermined time has passed, mirroring display is ended (S428), and if the predetermined time has not passed, the process proceeds to S426. If the device is not in mirroring display in S424, only remote control is started, continued, and ended based on user instruction information (S426).

[0065] If the data received from the linked device does not include user instruction information in S421, similarly to S422, it is confirmed whether the received data includes UI information (S430). If the received data includes UI information, only the start, continuation, and end of the mirroring display based on the UI information are executed (S431).

[0066] If the received data does not include UI information in S430, it is confirmed whether or not mirroring display is in progress, as in S424 (S432). If mirroring display is in progress in S432, it is determined whether or not the smart glasses 101 are in sleep mode (S433). If the smart glasses 101 are not in sleep mode in S433, mirroring display is continued (S434).

[0067] If it is determined in S433 that the device is in sleep mode, it is further determined whether or not a predetermined time has elapsed since the UI information was last received from the linked device (smart watch 110) (S435). If the predetermined time has elapsed, the mirroring display is ended (S436). If the predetermined time has not elapsed, the process proceeds to S434. If it is determined in S432 that the mirroring display is not in progress, the flow in FIG. 5 is ended.

[0068] 6 to 10 show examples of display on the "display 222 of the smart watch 110" and the "display 202 of the smart glasses 101" when executing a coordinated function of remote control and mirroring display. FIGS. 6 to 10 sequentially show a series of UI operations including AR display, assuming a use case in which a user starts remote control of the smart glasses 101 from the smart watch 110 and performs a web search using a web browser application on the smart glasses 101. Hereinafter, an application is abbreviated as "app."

[0069] 6A, 500 and 504 are initial screens of the smart watch 110 and the smart glasses 101, respectively. 501, 502, and 503 are all application icons on the smart watch 110, and are respectively a smart watch setting application, a web browser application, and a remote control application that controls applications on the smart glasses 101.

[0070] In addition, 505, 506, and 507 are all icons that are realized by being projected into the user's field of view by AR technology of the application executed by the smart glasses 101. They are icons for an application for setting the smart glasses, a web browser application, and a remote control application having a home appliance remote control function, respectively. In addition, applications other than the above-mentioned applications shown on the screens of 500 and 504 can also be displayed as application icons. When the smart glasses 101 recognize the user's finger and detect a selection operation of any of the icons provided by AR, the selected application is started and the content of the application is further displayed in the user's field of view. In the following, remote control by the smart watch 110 will be described, but it is also possible to use these processes in combination with an operation in response to the recognition of an input from the user in the smart glasses 101.

[0071] Here, when the remote control application 503 is pressed in Fig. 6(A), a remote control setting screen in Fig. 6(B) 508 is displayed. This screen has a control enable button 509 for enabling and starting remote control, and a target application selection button 509 for transitioning to a control target application selection screen when controlling an application on the smart glasses 101. The control enable button 509 is initially OFF, and pressing the control enable button 509 slides the ON / OFF toggle button to switch to control ON, as shown in Fig. 6(B).

[0072] When the control enable button 509 is switched to control ON, if the smart watch 110 and the smart glasses 101 have not yet been paired, a pop-up screen 511 for when pairing has not been performed is displayed as shown in FIG. 6C. The pop-up screen 511 for when pairing has not been performed displays a list 512 of candidates for the smart glasses 101 that can be paired, along with a warning message indicating that pairing has not yet been performed. In the example of the pop-up screen 511 for when pairing has not been performed, the candidate list 511 indicates that the smart glasses 101 named "Glasses-1" and "Glasses-2" can be paired by short-distance communication. Then, when the user selects "Glasses-1", an authentication screen (not shown) for pairing is displayed, and the user enters an authentication ID, an authentication password, and the like, and authentication and pairing are completed. To close the pop-up screen 511 for when pairing has not been performed, a Close button 513 is pressed.

[0073] If pairing has already been established when control is turned on using the control enable button 509 in FIG. 6B, or if pairing has been established using the pop-up screen 511 when pairing has not yet been established in FIG. 6C, remote control begins.

[0074] 7(A) shows a UI display state displaying a target application selection screen 514 for specifically controlling an application on the smart glasses 101 to be controlled during remote control. A transition to this state occurs when the user presses the above-mentioned target application selection button 510 in FIG. 6(B). A setting control application 515, a web browser control application 516, and a remote control control application 517 displayed on the target application selection screen 514 are control application icons for controlling the applications 505 to 507 on the smart glasses 101, respectively.

[0075] When a selection operation of the Web browser control application 516 on the smart watch 110 by the user is detected, the Web browser control screen 519 in Fig. 7(B) is displayed on the smart watch 110. Also, a Web browser screen 528, which is the screen displayed after the Web browser application 506 to be controlled is started, and a control in progress display 527, which indicates that remote control is being performed on the smart glasses 101, are displayed on the smart watch 110. As described above, the display on the smart watch 110 is a projection display into the user's field of vision using AR technology.

[0076] In-control display 527 displays the name of the linked terminal instructing the remote control so that it is clear from which linked terminal the remote control is being controlled, and in this example, it shows that it is being controlled by Watch-1. Web browser screen 528 is a search screen. Web browser screen 528 has a back button 528 for returning to the screen before the search after a search on the search screen, an input field 530 for inputting a search term, and a Click button 531 for executing a search with the term input in input field 530. Furthermore, Web browser screen 528 has a pointer 532 for moving and specifying an area on the Web browser screen, a browser update button, a button for minimizing / maximizing the browser display, a browser close button, a URL display field, and the like.

[0077] A back button 520 on a Web browser control screen 519 on the smart watch 110 corresponds to the above-mentioned back button 529, and is an instruction button for executing the back button 529. A BACK button 524 on the Web browser control screen 519 is a button for terminating the Web browser control application 516 and returning to the previous screen, the target application selection screen 514. When an END button 525 is pressed, the Web browser control application 516 and the remote control application 503 are terminated, the screen is closed, and the initial startup screen 500 is displayed.

[0078] The Web browser control screen 519 is an instruction screen for instructing control of the Web browser application 506 on the smart glasses 101, and this screen further includes an instruction control application whose functions are broken down into specific operation instructions.

[0079] The instruction control apps include, for example, a pointer movement instruction control app 521, a keyboard instruction control app 522, and a touchpad instruction control app 523. The pointer movement instruction control app 521 instructs the pointer 532 to move in a cross direction, and the keyboard instruction control app 522 instructs text input. The touchpad instruction control app 523 can give instructions equivalent to those of the touchpad function provided in conventional notebook PCs. Other control instruction apps include a scroll app that issues scroll instructions, a pinch-in / out app that issues pinch-in / pinch-out instructions, a flick app that slides the screen up, down, left, and right with a flick, and a tap app specialized for tap instructions.

[0080] Here, the glass linkage information for each application in Table 1 for the applications on the smart watch 110 shown in FIG. 6 and FIG. 7 is specifically shown in Table 2.

[0081] [Table 2]

[0082] The start of the remote control in FIG. 6 described above is executed based on the fact that the remote control application 503 (row of application ID: 0004) in Table 2 is remote control ON. Meanwhile, at this time, the mirroring display is OFF, so the mirroring display is not performed at this point. When the remote control application 503 is activated, the mirroring display may be defaulted to ON, and in that case, the remote control setting screen 508 is mirrored and displayed on the smart glasses 101 at the timing of the start of the remote control in FIG. 6.

[0083] In addition, after pressing the Web browser control application 516 in Fig. 7(A), the screen is displayed as shown in Fig. 7(B) as described above, but even in this case, remote control is being performed but mirroring display is not performed. This state is based on the fact that the Web browser control application 516 (row of application ID: 0006) in Table 2 is preset as remote control ON and mirroring display OFF. When mirroring display is preset as ON in the Web browser control application 516, the Web browser control screen 519 is mirrored and displayed on the smart glasses 101 at the timing shown in Fig. 7(B).

[0084] Fig. 8 shows a display example when a control application that performs mirroring display is executed, which transitions when the pointer movement instruction control application 521 is selected in Fig. 7(B). Fig. 8(A) shows a display example when mirroring display is started.

[0085] In Table 2, the pointer movement instruction control application 521 (row of application ID: 0008) has remote control ON and mirroring display ON, and based on this default, remote control continues and mirroring display starts.

[0086] In the smart watch 110, a pointer movement instruction screen 533 is displayed, and it is possible to give instructions to move the pointer up, down, left, and right using a cross key 534. In addition, a tap instruction can be given by pressing a TAP button in the center of the cross key 534. In addition, a BACK button 535 is a button for ending the pointer movement instruction control application 521 and returning to the previous screen, the Web browser control screen 519. When the END button 535 is pressed, the pointer movement instruction control application 521, the Web browser control application 516, and the remote control application 503 are ended, the screen is closed, and the initial startup screen 500 is displayed.

[0087] In FIG. 8(A), the wording "mirroring display in progress 536" indicating that mirroring display is in progress is additionally displayed in the user's field of vision through the smart glasses 101. This display displays the name of the associated terminal so that it is possible to know which associated terminal's UI is being mirrored, and in this case, it indicates that the UI of Watch-1 is being displayed. 537 is a mirroring display image, and at this timing, the UI of the pointer movement instruction control screen 533 displayed on the smart watch 110 of Watch-1 is mirrored and displayed. Also, the display position of the Web browser screen 528 in FIG. 8(A) has been moved and the screen size reduced from that of FIG. 6(I) before mirroring display. Note that in this example, such automatic control is added in order to secure an area for displaying the above-mentioned mirroring display image 537.

[0088] 8(A), when the user operates the down arrow of the cross key 534 on the smart watch 110, the mirroring display image 537 reflects the operation, and the pointer 523 on the web browser screen 528 moves down according to the instruction. By reflecting the operation content accepted by the smart watch 110 on the mirroring display image 537, the user can visually confirm his / her own operation again, and duplicate operations and erroneous operations can be prevented. In FIG. 8(A), it is shown that the pointer has been moved to the input field 530.

[0089] Next, when the user presses the TAP button in the center of the cross key 534 in FIG. 8(B), the operation is similarly reflected in the display of the mirroring display image 537, and since the pointer is on the input field 530, the input field 530 is selected.

[0090] 8, when the user presses BACK button 535, the screen displayed at the time of pressing BACK is also reflected in mirroring display image 537, and the screen returns to Web browser control screen 519 in Fig. 7(B). Then, pointer movement instruction control application 521 is terminated, and Web browser control application 516 is being operated, so that mirroring display is terminated (OFF) according to the information of application ID "0006" in Table 2.

[0091] In the case of a simple operation unit as shown in the mirrored display image 537 in this figure, the user may be able to input operations to the smartwatch 110 while checking the mirrored display image 537 without looking at the display of the smartwatch 110. Therefore, the user will be able to control the AR app with only slight operations at hand, without having to look away, such as lowering his / her face toward the smartwatch 110.

[0092] FIG. 9 is a display example different from FIG. 8 when a control application that performs mirroring display is executed, which transitions when the keyboard input instruction control application 522 is selected in FIG. 7(B), and mirroring display is started as in FIG. 9(A). The smart watch 110 displays a keyboard input instruction control screen 538. The keyboard input instruction control screen 538 displays a keyboard 539 and a BACK button 540 for closing the keyboard input instruction control application 521 and returning to the previous screen, the Web browser control screen 519. At this time, since the keyboard input instruction control application 522 is being operated, mirroring display is started (ON) again according to the information of the application ID "0008" in Table 2, and a mirroring display image 541 is displayed. The mirroring display image 541 is a mirroring display of the UI of the keyboard input instruction control screen 538 displayed on the smart watch 110 of Watch-1 at this timing.

[0093] 9(B), when the user selects the character "ya" on keyboard 539, for example, input candidate characters beginning with "ya" are displayed in candidate character list 542. When the user selects a character from candidate character list 542, confirmation button 543 for confirming the selection is also displayed. Then, the UI of keyboard input instruction control screen 538 at this timing is displayed and reflected in mirroring display image 541, and the selected character "ya" is displayed in input field 530 on Web browser screen 528.

[0094] For example, when the user selects "mountain" from the candidate characters in the candidate character list 542 of FIG. 9(B), it is displayed and reflected in the mirroring display image 541, and "mountain" is displayed in the input field 530 on the Web browser screen 528 in a state where it is being selected as a candidate. In this state, when the user presses the confirmation button 543 in FIG. 6(S), it is similarly displayed and reflected in the mirroring display image 541, and "mountain" in the input field 530 on the Web browser screen 528 is also displayed in a confirmed state and waits for the input of the next character.

[0095] When the BACK button 540 is pressed in FIG. 9, the process returns to the Web browser control screen 519 of FIG. 7(B). At this timing, the keyboard input instruction control application 522 ends and the Web browser control application 516 is in operation. Therefore, according to the information of the application ID "0006" in Table 2, the mirroring display is terminated (turned off).

[0096] FIG. 10 is a display example when the control application that does not perform mirroring display is executed, which is a transition when the touchpad instruction control application 523 is selected in FIG. 7(B). As shown in FIG. 10(A), the application on the Web browser screen 528 is controlled without mirroring display. On the smartwatch 110, a touchpad instruction control screen 544 is displayed. On the touchpad instruction control screen 544, a touchpad area 545 for touchpad instruction and a BACK button 546 for ending the touchpad instruction control application and returning to the previous Web browser control screen 519 are displayed. The touchpad instruction control application 523 continues remote control with the mirroring display turned off according to the information of the application ID "0010" in Table 2. Therefore, there is no change in the display on the smart glass 101 during the state transition from FIG. 7(B) to FIG. 10(A). When the user moves a finger to the right in the touchpad area 545 of FIG. 10(A), the pointer 532 on the Web browser 528 also moves to the right in conjunction with the movement. Here, it is assumed that the pointer has moved up to the Click button 531.

[0097] Next, when the user taps twice on touchpad area 545 in FIG. 10(B), the movement causes a pointer 532 on web browser 528 to click on Click 531, which is specified by the tap. This causes a web search to be performed for the word "mountain" entered in input field 530. FIG. 10(C) shows the screen after the search operation, with search results displayed as in web browser screen 528. On web browser screen 528, the number of hits, link information to the hit articles, image information, etc. are displayed in a list in search result information area 547.

[0098] When the BACK button 546 in FIG. 10 is pressed, the screen returns to the Web browser control screen 519 in FIG. 7(B).

[0099] By using the UI operations shown in FIGS. 6 to 10, when remotely controlling an application on the smart glasses 101 from the smart watch 110, it is possible to turn the mirroring display ON / OFF according to the application being operated.

[0100] Here, in a series of UI operations as shown in FIGS. 6 to 10, even when the smart glasses 110 are remotely controlled from the smart watch 110, as described above, the smart glasses 110 can accept user input instructions via the operation unit 304 or the like. Therefore, while the UI of the smart watch 110 is being mirrored and displayed, a user operation via the operation unit 304 of the smart glasses 110 may interrupt. In this case, the smart glasses 110 once end the mirroring display. In this case, when the smart watch 110 detects an application operation for turning on the mirroring display in display 2 again and instructs the smart glasses 110 to perform mirroring, the smart glasses 110 resumes mirroring.

[0101] 6 to 11, the user selects the control target application selection button 510 in Fig. 6(B) and then presses the Web browser control application 516 in Fig. 7(A) to start mirroring display. In other words, this is a use case in which the application management unit 328 on the smartwatch 110 receives a notification of "user input from the operation unit 325" and detects operation information for the Web browser control application 516 on the smartwatch 110 (S401).

[0102] On the other hand, there is another use case in which the application management unit 328 on the smart watch 110 receives a "control instruction application launch request from the smart glasses 101" via the communication unit 301 and launches the control instruction application on the smart watch 110. At this time, the smart watch 110 that has received the notification launches the control instruction application and detects the launch of the control instruction application (S401). This other use case will be briefly described with reference to FIG. 6.

[0103] First, in this use case, the user does not select the control target application selection button 510 in FIG. 6(B), but instead presses the Close button (not shown) on the remote control setting screen 508, causing remote control to be performed in the background, and the initial screen 500 remains displayed.

[0104] Next, during the waiting state on this initial screen 500, for example, the user uses the operation unit 304 on the smart glasses 101 to select an input field 530 on the Web browser screen 528 being displayed on the smart glasses 101. Then, the application management unit 309 detects the "selection of the input field," and the application management unit 309 requests the smart watch 110 via the communication unit 301 to "launch the keyboard input instruction control application 522 shown in FIG. 6."

[0105] Then, upon receiving a request to start the keyboard input instruction control application 522, the smart watch 110 starts the keyboard input instruction control application 522 and displays a keyboard input instruction screen 538. The user inputs data using the displayed keyboard input instruction screen 538.

[0106] After that, when the user finishes inputting characters and presses the Close button (not shown) on the keyboard input instruction screen 538, the smartwatch 110 terminates the keyboard input instruction control application 522 and the smartwatch 110 returns to the waiting state on the initial screen 500 again.

[0107] According to the embodiment described above, it is possible to improve the usability in collaboration when remotely controlling the functions of a smartglasses application with a smartwatch by using AR technology.

[0108] In this embodiment, the description is given under the name of mirroring display, but it is not necessary for the smart glasses 101 to project a display image (for example, the mirroring display image 537 in FIG. 8A) that completely matches the operation unit and operability displayed on the smart watch 110. It is sufficient to realize a virtual display of the operation unit to the extent that the operability can be grasped, and the mirroring display may be realized with any image showing the operation unit customized by the user, the application provider, the smart glasses 101, etc.

[0109] <Embodiment 2> There are various usage environments when a user uses the smart watch 110 and the smart glasses 101 together. Therefore, by taking into consideration the usage environment and environmental changes when these devices are linked, it is possible to further improve usability for the user when they are linked.

[0110] In the second embodiment, an embodiment that takes into consideration the usage environment and environmental changes during collaboration will be described by showing the differences from the first embodiment.

[0111] In the software module configuration of the smart watch 110 of this embodiment (FIG. 3(B)), the application management unit 328 makes it possible to set in advance, as per-application glass linkage information, which functions are enabled in which usage environment when linking with the smart glasses 101. It is also possible to set the functions to be activated when a change in the user's usage environment is detected. These settings are made for each of various applications on the smart watch 110. As an example, they are preset as shown in Table 3.

[0112] [Table 3]

[0113] In the glass collaboration based on the glass collaboration information for each application in Table 3, it is predefined in which usage environment the remote control and mirroring display described in the first embodiment are enabled. In the second embodiment, not only is the "ON / OFF setting" that sets whether the function is enabled or disabled set, but the usage environment when it is enabled (ON) is also set as the "usage environment setting."

[0114] For example, taking the application with application ID "0001" in Table 3 as an example, first the remote control ON / OFF setting is "ON" and the usage environment setting is "Crowded train or walking". In this case, if it is detected that the user environment is "Crowded train" or "Walking", the remote control function is started. After that, if it is detected that the user environment is neither "Crowded train" nor "Walking", the remote control function is terminated. The mirroring display ON / OFF setting for application ID "0001" is also "ON" and the usage environment setting is "Crowded train". In this case, if it is detected that the user environment is "Crowded train", the mirroring display function is started. Also, if it is detected that the user environment is not "Crowded train", the mirroring display function is terminated.

[0115] Here, the application management unit 309 detects an environmental change, such as whether the user environment is a "crowded train" or "walking," based on information from the sensor unit 209 and the camera 210 of the smart glasses 110. The application management unit 309 also manages the environmental status indicating the current user environment in the ROM 203 and ROA 204 via the storage unit 302. When the smart glasses 101 detect an environmental change in the application management unit 309, it notifies the smart watch 110. When the smart watch 110 receives the environmental change detection, it checks its own application in operation. If there is an application in operation, it turns on or off "remote control" and "mirroring display" based on the glass cooperation information for each application in Table 3.

[0116] Some applications are specified in the usage environment settings of Table 3 with "- (hyphen)", which indicates that the application is not specified. In this case, the application follows the setting of "ON / OFF setting" in all environments. In addition, some applications specify "direct viewing of linked screen" in the usage environment settings of Table 3. This indicates a state in which the user is looking at the screen (display 222) of the smart watch 110 that is physically linked through the lens of the smart glasses 101. As in the application ID "0003" in Table 3, the ON / OFF setting is OFF in the mirroring display, and the usage environment setting is "direct viewing of linked screen". In this case, when the user looks directly at the screen of the smart watch 110 during mirroring display on the smart glasses 101, the mirroring display ends, and when the user takes his / her eyes off the screen of the smart watch 110, the mirroring display starts again. This relieves the user from the hassle of double display of the same screen, such as the same screen being mirrored and displayed on the smart glasses 101 when the user is looking directly at the screen of the smart watch 110.

[0117] Furthermore, as mentioned above, if the usage environment settings are set to "crowded train or walking," it indicates "crowded train" or "walking." As a similar setting, although not mentioned in the example of Table 3, if the usage environment settings are set to "looking directly at the linked screen and walking," it indicates "looking directly at the linked screen" and "walking."

[0118] As described above, the definition of "environment" in the second embodiment is not limited to the environment in which the user is located, "crowded train," as shown in the usage environment settings in Table 3, but also includes user situations such as "walking" and "looking directly at the linked screen." Also, in Table 3, an example is given in which values ​​such as "crowded train" and "walking" are managed in the "usage environment settings," but this is for the sake of explanation. In reality, the setting value IDs assigned to each of these environments may be managed in a table.

[0119] Furthermore, in the second embodiment, a watch screen lock function is also provided to lock the screen of the smart watch 110 when a specific environmental change is detected. This is set by providing columns of "ON / OFF setting" and "activation environment" in the "watch screen lock" column in Table 3. For example, in the application ID "0001" in Table 3, the ON / OFF setting in the "watch screen lock" column is "ON", and the activation environment is "crowded train". In this case, it is indicated that the watch screen lock function is executed when it is detected that the user environment has transitioned from "environment other than a crowded train" to "environment of a crowded train". This watch screen lock function is used to reduce the possibility that another user touches the screen (display unit 111) of the smart watch 110 in a crowded train state, causing erroneous control in remote control of the smart glasses 101. If the screen of the smart watch 110 is locked once when the train is crowded, erroneous control due to another user's touch can be prevented.

[0120] After the screen is locked using the watch screen lock function, it is possible to ensure that the user can operate the screen with their own touch by, for example, having the user log in again using fingerprint authentication on the screen.

[0121] Another effective use of the watch screen lock function is to lock the screen of the smart watch 110 when the state transitions to "walking." When the user remotely controls the smart watch 101 from the screen of the smart watch 110, there are cases where the user looks directly at the screen of the smart watch 110. In this state, when the user starts walking, the user walks while looking directly at the screen of the smart watch 110, and the user cannot notice even if there is a dangerous environment around. Therefore, by temporarily locking the screen when the state transitions to "walking," it is possible to provide an opportunity for the user to temporarily turn away from the screen of the smart watch 110 and turn their attention to the surrounding environment. When the screen is locked by the watch screen lock function at this time, a warning may be displayed on the screen.

[0122] In addition, when the screen is unlocked again while the "walking" state continues, the user may be prompted to display the mirrored screen. In this case, the series of processes from displaying a warning when the screen is locked to displaying the mirrored screen after the screen is unlocked may be included in the "watch screen lock" function.

[0123] Furthermore, as a means for releasing the screen lock by "Watch Screen Lock", a "Watch Screen Release" function not listed in Table 3 may be set as one of the linkage functions. For example, one method is to specify an environment indicating "Stopped (Walking OFF)" in the usage environment settings in Table 3, and set the ON / OFF setting of "Watch Screen Release" to ON.

[0124] 11 to 13 are flowcharts showing a process that characterizes the present embodiment 2 when the smart watch 110 controls an application installed in the smart glasses 110 in the pairing between the smart watch 110 and the smart glasses 101. In addition, in Figs. 11 to 13, it is assumed that the smart watch 110 and the smart glasses 101 have already been paired by the pairing unit 327 and the pairing unit 308, respectively.

[0125] 11 is a flowchart of a process of transmitting a notification of an environmental change to the smart watch 110 when a change in the user environment is detected in the smart glasses 101. The application management unit 309 is the subject of the processes included in this flowchart.

[0126] The application management unit 309 detects a change in the environment around the user based on information from the sensor unit 209 and the camera 210 (S601). For example, the sensor unit 209 detects whether or not the user is walking based on information from a GPS or an acceleration sensor. Also, based on image information from the proximity sensor of the sensor unit 209 and the camera 210, it detects whether or not the user is in a crowded train. Furthermore, if the camera 210 has a function of capturing an object at approximately the same eye level as the user of the smart glasses 101, it detects whether or not the user is directly looking at the screen of the linked device (smart watch 110) in linkage based on image information from the camera 210. The above detection method is one example, and detection is performed based on the functions of the smart glasses 101.

[0127] 11 if no environmental change is detected in S601, and stores the environmental change as the latest environmental information in the RAM 204 via the storage unit 302 if an environmental change is detected. Then, the application management unit 309 notifies the smart watch 110 of the detected environmental information (S603).

[0128] Here, the latest environmental information is stored in a table such as Table 4.

[0129] [Table 4]

[0130] The application management unit 309 predefines environmental items that it can detect, such as those in the "Environment" column of Table 4. For example, if the environment changes from a standing still state to walking, the "walking" state is updated to ON. After that, if the environment changes from walking to a standing still state, the "walking" state is updated to OFF.

[0131] 12 is a flowchart of a process in which, when the smart watch 110 receives environmental information from the smart glasses 101, the smart watch 110 transmits control information for remote control or mirroring display determined based on the received information to the smart glasses 101. The application management unit 328 is the subject of the process included in this flowchart.

[0132] First, in S611, if the application management unit 328 receives environmental information from the smart glasses 101, the process proceeds to S612, and if not, the process proceeds to S613. In S612, the application management unit 328 stores the latest environmental information (Table 4) received in the RAM 224 via the storage unit 323. S613 is the same as S401, and if there is an application operation, the process proceeds to S614, and if there is no application operation, the process proceeds to S621.

[0133] Here, S621 and S622 are equivalent to S408 and S409, respectively. Also, S623 is equivalent to a process combining S410 and S411. In S623, if the above-mentioned per-application glass linkage information (Table 3) specifies linkage functions to be executed by the smart glasses 101 other than the remote control and mirroring functions, all of those linkage functions are also terminated. The smart glasses 101 are also notified of the result of this termination.

[0134] In S614, the latest environment information (Table 4) stored in S612 is acquired, and if there is a "linked function to be executed on the own terminal" that matches the default in the usage environment setting in Table 3 based on the environment in which the state in Table 4 is ON (S615), the linked function is executed on the own terminal (S616). In this case, the "linked function to be executed on the own terminal" corresponds to the "watch screen lock" function in Table 3 as an example. The watch screen lock function is a function that locks the screen of the smart watch 110, and therefore is a function that should be executed on the own terminal. As described above in Table 3, the "watch screen lock" function may include a warning display during locking and a mirroring display execution after unlocking. In addition, the "watch screen release" function, also described above in Table 3, is also included in the "linked function to be executed on the own terminal".

[0135] Next, in S617, based on the environment in which the state in Table 4 is ON, if there is a "linked function to be executed by smart glasses" that matches the default in the usage environment setting in Table 3 (S617), the status information of the relevant linked function is notified (S618). Here, the "linked function to be executed by smart glasses" corresponds to the functions of "remote control" and "mirroring display" in Table 3 as an example. The remote control function controls an application on the smart glasses 101. Also, the mirroring display performs mirroring display on the smart glasses 101. Therefore, these functions are functions that need to be executed by the smart glasses 101. Also, the status information of the linked function notified in S618 is, for example, the result of determining whether the remote control or mirroring display should be ON or OFF based on Table 4. Also, at the time of this notification, information such as the application ID and application name of the application whose operation was detected in S613 is also included in the information and notified.

[0136] So far, we have explained the functions to be executed on the own terminal and the functions to be executed on the smart glasses, which are set in Tables 3 and 4. However, the information to determine which is the target link function may be set by adding an attribute to the glass link information for each application (Table 3), or may be managed in a separate table.

[0137] In S619, it is determined whether or not there is a function in cooperation with the smart glasses 101. This determination is made, similarly to S617, based on an environment in which the state of Table 4 is ON, if there is a "cooperative function to be executed by the smart glasses" that matches the default in the usage environment setting of Table 3 based on the environment in which the state of Table 4 is ON, and if there is a function that should be turned ON, it is determined that there is a function in cooperation. In addition to this determination method, the notification history of past state information in S618 may be stored in the storage unit 323, and it may be determined whether or not there is a function in cooperation based on the past notification history. In S619, if there is a function in cooperation with the smart glasses 101, proceed to S620, and if there is no function in cooperation with the smart glasses 101, the flowchart shown in FIG. 12 is terminated.

[0138] S620 corresponds to the processes of S404, S405, and S407 in Fig. 4. For example, if cooperation is being performed using the remote control function, user instruction information is transmitted as application operation information. Also, if cooperation is being performed using mirroring display, UI information is transmitted as application operation information.

[0139] 13 is a flowchart of the processing performed by the smart glasses 101 when "state information for turning on / off a link function" or "application operation information corresponding to a link function in linkage" is received from the smart watch 110. The subject of the processing included in this flowchart is c.

[0140] If the application management unit 309 receives "linked function status information" from the linked device (smart watch 110) in S631, the process proceeds to S632, and if not, the process proceeds to S633. The "linked function status information" received in S631 is the information notified from the smart watch 110 in S618 or S623 of FIG.

[0141] In S632, this received information is reflected as a process to be executed by the smart glasses 101. Specifically, when remote control ON is received, control of an application on the smart glasses 101 that can be controlled by the application ID on the smart watch 110 included in the notification information of S618 is started. When remote control OFF is received, control of this application is ended. When mirroring display ON is received, the application management unit 309 starts a mirroring display application on the smart glasses 101, thereby starting mirroring display. When mirroring display OFF is received, mirroring display is ended.

[0142] In S633, if "application operation information corresponding to the linked function being linked" is received from the smart watch 110, proceed to S634, and if not, proceed to S635. The "application operation information corresponding to the linked function being linked" is the information notified from the smart watch 110 in S620 of FIG.

[0143] In S634, this received information is reflected as a process executed by the smart glasses 101. Specifically, when the cooperation function during cooperation is remote control, control based on user instruction information as application operation information is reflected on the application on the smart glasses 101. Also, when the cooperation function during cooperation is mirroring display, the application operation information becomes UI information corresponding to information of display content provided in the mirroring display on the smart glasses 101. Here, for example, the display content changed by the user operating the application on the smart watch 110 is UI information, and the mirroring display is updated based on the UI information.

[0144] Since S635 to S639 are equivalent to the processes of S424 to S428, respectively, a description thereof will be omitted.

[0145] According to the above flowchart, it is possible to improve usability by taking into account the user's usage environment when coordinating remote control of smart glasses functions using a smart watch.

[0146] In the second embodiment, the smart glasses 101 detects an environmental change and notifies the smart watch 110 (FIG. 11), but environmental change detection may be performed by another configuration. For example, the smart watch 110 itself may detect an environmental change using the application management unit 328 based on information from a sensor unit 229 mounted thereon. In addition, the smart glasses 101 or the smart watch 110 may be provided with various environmental modes such as a normal mode, a crowded train mode, and a walking mode, and the user may be able to select which environmental mode to use. The user may then switch between the environmental modes as an alternative means for detecting an environmental change.

[0147] In the examples of the first and second embodiments described above, the configuration of each wearable device, the configuration of a software module, and a flowchart related to application notifications of the wearable device have been shown, but these are merely examples and are not limiting.

[0148] According to the present invention including each of the above-mentioned embodiments and combinations thereof, a high-performance operation unit for handling various operations required for various applications is not required for smart glasses. Therefore, it is possible to remove the design constraints caused by mounting many such operation units on smart glasses. In other words, the present invention can be said to greatly contribute not only to improving usability but also to improving the design of smart glasses, and may also contribute to weight reduction.

[0149] (Other Examples) The present invention also includes an apparatus or system configured by appropriately combining the above-described embodiments, and a method thereof.

[0150] Here, the present invention is a device or system that executes one or more pieces of software (programs) that realize the functions of the above-mentioned embodiments. Also, a method for realizing the above-mentioned embodiments executed by the device or system is also one aspect of the present invention. Also, the program is supplied to the system or device via a network or various storage media, and the program is read into one or more memories by one or more computers (CPU, MPU, etc.) of the system or device and executed. In other words, as one aspect of the present invention, the program itself, or various storage media that store the program and can be read by a computer, are also included. Also, the present invention can be realized by a circuit (e.g., ASIC) that realizes the functions of the above-mentioned embodiments. [Explanation of symbols]

[0151] 101 Smart Glasses 110 Smart Watch

Claims

1. A glasses-type information device worn by a user having a linked device, a first control means for controlling a display such that an image corresponding to an operation unit for remotely controlling an application executed on the glasses-type information device by the cooperative device is projected in the user's field of vision; a second control means for controlling a function of the application based on information for the application received in response to an operation of the user performed on the cooperative device, The glasses-type information device is capable of executing one or more applications; the first control means, when an application for which a setting for displaying an image corresponding to the operation unit is enabled is being executed on the glasses-type information device, projects an image corresponding to an operation unit related to a remote control of the application into a field of vision of the user via the glasses-type information device; When an application in which a setting for displaying an image corresponding to the operation unit is disabled is executed on the glasses-type information device, the first control means does not project, via the glasses-type information device, an image corresponding to the operation unit for remote control of the application. A glasses-type information device characterized by:

2. A glasses-type information device worn by a user having a linked device, a first control means for controlling a display such that an image corresponding to an operation unit for remotely controlling an application executed on the glasses-type information device by the cooperative device is projected in the user's field of vision; a second control means for controlling a function of the application based on information for the application received in response to an operation of the user performed on the cooperative device; a detection unit that detects information for identifying an environment related to control of a display of an image corresponding to an operation unit related to a remote control of the application by the first control unit, The first control means projects an image corresponding to an operation unit for remote control of an application executed on the glasses-type information device into the user's field of vision via the glasses-type information device in accordance with a setting related to an environment specified by the information detected by the detection means. A glasses-type information device characterized by:

3. The glasses-type information device according to claim 1 , wherein the image corresponding to the operation unit is an image based on information received from the linked device.

4. The glasses-type information device according to any one of claims 1 to 3, characterized in that the first control means further controls display so as to reflect the user's operation on an image corresponding to the operation unit, based on information for the application received in response to the user's operation performed on the linked device.

5. 5. The glasses-type information device according to claim 1, wherein the glasses-type information device is one of smart glasses and a head-mounted display.

6. 6. The glasses-type information device according to claim 1, further comprising a network module for connecting to the Internet.

7. A method for a glasses-type information device worn by a user having a linked device, comprising: a first control step of controlling a display so that an image corresponding to an operation unit related to a remote control by the cooperative device for an application executed on the glasses-type information device is projected in the field of view of the user; a second control step of controlling a function of the application based on information for the application received in response to an operation of the user performed on the linked device; The glasses-type information device is capable of executing one or more applications; In the first control step, when an application in which a setting for displaying an image corresponding to the operation unit is enabled is executed on the glasses-type information device, an image corresponding to an operation unit for remote control of the application is projected into the user's field of vision via the glasses-type information device; When an application in which a setting for displaying an image corresponding to the operation unit is disabled is executed on the glasses-type information device, an image corresponding to the operation unit for remote control of the application is not projected via the glasses-type information device. A method comprising:

8. A method for a glasses-type information equipment worn by a user having a linked device, comprising: a first control step of controlling a display so that an image corresponding to an operation unit related to a remote control by the cooperative device for an application executed on the glasses-type information device is projected in the field of view of the user; a second control step of controlling a function of the application based on information for the application received in response to an operation of the user performed on the cooperative device; a detection step of detecting information for identifying an environment related to control of display of an image corresponding to an operation unit related to remote control of the application in the first control step, In the first control step, an image corresponding to an operation unit for remote control of an application executed on the glasses-type information device is projected into the user's field of vision via the glasses-type information device in accordance with a setting related to the environment identified by the information detected in the detection step. A method comprising:

9. A computer as a glasses-type information device worn by a user having a linked device, a first control means for controlling a display such that an image corresponding to an operation unit for remotely controlling an application executed on the glasses-type information device by the cooperative device is projected in the user's field of vision; a program for causing the program to function as a second control unit that controls a function of the application based on information for the application received in response to an operation of the user performed on the cooperative device, The glasses-type information device is capable of executing one or more applications; the first control means, when an application for which a setting for displaying an image corresponding to the operation unit is enabled is being executed on the glasses-type information device, projects an image corresponding to an operation unit related to a remote control of the application into a field of vision of the user via the glasses-type information device; When an application in which a setting for displaying an image corresponding to the operation unit is disabled is executed on the glasses-type information device, the first control means does not project, via the glasses-type information device, an image corresponding to the operation unit for remote control of the application. A program characterized by:

10. A computer as a glasses-type information device worn by a user having a linked device, a first control means for controlling a display such that an image corresponding to an operation unit for remotely controlling an application executed on the glasses-type information device by the cooperative device is projected in the user's field of vision; a second control means for controlling a function of the application based on information for the application received in response to an operation of the user performed on the cooperative device; A program for causing the first control means to function as a detection means for detecting information for identifying an environment related to control of display of an image corresponding to an operation unit related to remote control of the application, the program comprising: The first control means projects an image corresponding to an operation unit for remote control of an application executed on the glasses-type information device into the user's field of vision via the glasses-type information device in accordance with a setting related to an environment specified by the information detected by the detection means. A program characterized by:

11. The program described in claim 9 or 10, characterized in that the first control means further controls display so as to reflect the user's operation on an image corresponding to the operation unit based on information for the application received in response to the user's operation performed on the linked device.

12. The program described in any one of claims 9 to 11, characterized in that the glasses-type information device is either smart glasses or a head-mounted display.

13. The program described in any one of claims 9 to 12, characterized in that the glasses-type information device has a built-in network module for connecting to the Internet.

Citation Information

Patent Citations

  • Data input device and method

    JP2002244791A

  • Head-mounted display and information display apparatus

    JP2013125247A

  • Head-mounted displays and how to control them

    JP2016506530A

  • Cooperative display system

    JP2019036914A

  • Display system, controller, control method of display system and program

    JP2021119431A