Information processing terminal

Abstract

To provide a plurality of information terminals having the same applications or corresponding applications mounted thereon, and to easily start up and execute the applications with a desired device without increasing a burden on a user.SOLUTION: An information processing terminal (100) for executing applications provided in common in terminals in cooperation with another external terminal (200), comprises: an output unit (175) that outputs a result of execution of the applications executed in a cooperative operation mode that is an operation mode in which the information processing terminal and the other external terminal execute the applications in cooperation with each other; and a cooperation determination unit (173) that determines whether or not the other external terminal is in a cooperative state in which it is operable in the cooperative operation mode. The cooperative state is determined by the arrangement of the information processing terminal and the other external terminal, and when an icon that receives a start-up instruction displayed on a display of the other external terminal is included in a predetermined cooperative visual field of a display of the information processing terminal, the other external terminal is determined to be in the cooperative state.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention provides a method for launching applications (hereinafter referred to as apps) between multiple information terminals. It relates to control technology. [Background technology]

[0002] Head Mounted Display (HMD) However, because the display area is small, it is sometimes used in conjunction with other information terminals. Document 1 states, "Information used to connect to a linked device and information on the linked device A first management for managing device information including information regarding data used by registered apps. In response to a request for cooperation from a cooperation source device, the cooperation source device refers to the device information and determines whether cooperation is possible. a second management unit that acquires collaboration candidates indicating suitable collaboration destination devices and applications; Launch the app shown in and check the device you want to link and the link registered on that device. The application linking device includes a linking control unit that determines which applications are linkable with each other (summary excerpt) is shown. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-142693 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology of Patent Document 1, a desired application can be selected and specified using another information terminal. However, in the technology of Patent Document 1, the application does not have access to all registered information. Therefore, the application may be executed on unintended information devices, This may interfere with work on the information device, such as causing other apps to stop working.

[0005] The present invention has been made in consideration of the above circumstances, and is intended to provide a device that is compatible with the same or corresponding application. This allows users to easily access multiple information terminals on their desired device without increasing the burden on the user. To provide an easy-to-use information processing technology for starting and executing an application. [Means for solving the problem]

[0006] The present invention is an information processing terminal that cooperates with other external terminals to execute an application that is common to each terminal, the information processing terminal comprising: an output unit that outputs the execution result of the application executed in a cooperative operation mode, which is an operation mode in which the application is executed in cooperation between the information processing terminal and the other external terminal; and a cooperation judgment unit that judges whether the other external terminal is in a cooperative state in which it can operate in the cooperative operation mode, the cooperative state being determined by the arrangement of the information processing terminal and the other external terminal, and the cooperation judgment unit determining that the information processing terminal is in the cooperative state when an icon that accepts a startup instruction displayed on the display of the other external terminal is included within a predetermined cooperative field of view of the display of the information processing terminal. Also, there is provided an information processing terminal that cooperates with other external terminals to execute an application that is provided in common to each terminal, the information processing terminal comprising: a reception unit that receives an instruction to launch the application; and a cooperation control unit that operates the information processing terminal in either a standalone operation mode or a collaborative operation mode, the standalone operation mode being an operation mode in which the application is executed by the information processing terminal alone; and the collaborative operation mode being an operation mode in which the application is executed in collaboration with the other external terminal and the information processing terminal, the collaboration control unit determining whether the information processing terminal is in a collaborative state in which it can operate in the collaborative operation mode, and if it determines that it is in the collaborative state, operating the information processing terminal in the collaborative operation mode, and if the other external terminal is within a predetermined range of the information processing terminal, determining that it is in the collaborative state. [Effects of the Invention]

[0007] According to the present invention, user communication can be performed between a plurality of information terminals that are equipped with the same application or corresponding applications. It allows users to easily launch and run apps on their desired devices without increasing the burden on the user. It is possible to provide an information processing technology that is easy to use. The problems, configurations and effects other than those mentioned above are as follows: This will become clear from the description of the embodiments below. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an information processing system according to a first embodiment. [Figure 2] 5A and 5B are explanatory diagrams for explaining a linked state in the first embodiment. [Figure 3] 5A and 5B are explanatory diagrams illustrating a non-cooperative state in the first embodiment. [Figure 4] FIG. 2 is a hardware configuration diagram of the head-mounted display according to the first embodiment. [Figure 5] FIG. 2 is a hardware configuration diagram of the smartphone according to the first embodiment. [Figure 6] 1A and 1B are functional configuration diagrams of a head-mounted display and a smartphone, respectively, according to a first embodiment. [Figure 7] 10 is a flowchart illustrating an example of application cooperation processing according to the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of application cooperation processing according to a second embodiment. [Figure 9] FIG. 11 is an explanatory diagram for explaining a cooperation status determination method according to a third embodiment. [Figure 10] 13 is a flowchart illustrating an example of application cooperation processing according to the third embodiment. [Figure 11] 10A and 10B are diagrams illustrating the display content of a display of a smartphone according to a fourth embodiment. [Figure 12] 13 is an example of a flowchart of application cooperation processing according to the fourth embodiment. [Figure 13] 13 is a flowchart illustrating an example of application cooperation processing according to the fifth embodiment. [Figure 14] 13 is a flowchart illustrating an example of application cooperation processing according to the sixth embodiment. [Figure 15] FIG. 13 is an explanatory diagram illustrating an overview of an information processing system according to a seventh embodiment. [Figure 16] 13(a) and 13(b) are explanatory diagrams for explaining a linked state in the seventh embodiment. [Figure 17] FIG. 13 is a hardware configuration diagram of a smart watch according to a seventh embodiment. [Figure 18] FIG. 13 is a functional block diagram of a smart watch according to a seventh embodiment. [Figure 19] 13 is a flowchart illustrating an example of application cooperation processing according to the seventh embodiment. [Figure 20] 10(a) and 10(b) are explanatory diagrams for explaining an overview of an information processing system according to a modified example of the present invention. [Figure 21] 10(a) to 10(e) are explanatory diagrams for explaining icon display modes according to modified examples of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] <<First Embodiment>> A first embodiment of the present invention will be described with reference to the drawings. The application shared by the first information terminal and the second information processing terminal applications (apps) are run in collaboration with each other.

[0010] Hereinafter, this embodiment will be described with reference to a head-mounted display (hereinafter referred to as a head-mounted display) as the first information terminal. , referred to as HMD) as the second information terminal, and a smartphone (hereinafter referred to as smartphone). An example of how to use this will be explained below.

[0011] FIG. 1 is a diagram schematically illustrating the entire information processing system 500 of this embodiment. As shown in FIG. 1, the information processing system 500 of this embodiment is worn on the head of a user 501. The first information terminal is an HMD 100 held by a user 501, and the second information terminal is a smartphone held by a user 501. It is equipped with a ho200.

[0012] The HMD100 creates a virtual world that resembles reality, giving you the feeling that you are actually there. Virtual reality (VR) technology allows people to experience the real world digitally. Virtual space (virtual space) created by CG (Computer Graphics) etc. with added information Augmented Reality (AVR) is a technology that reflects and extends imaginary objects in the real world. It is an information terminal that embodies AR (augmented reality) technology, and is worn by a user 501 on the head. However, due to structural constraints, the display range is narrow and it is difficult to perform detailed operations. Difficult.

[0013] The smartphone 200 has a function for making calls and communicating independently. The smartphone 200 has an icon. The icons are displayed on the screen in an easy-to-read format and are suitable for touch operation. The application can be started by the user 501 approaching or touching the device with a finger or a touch pen. It is a useful graphic symbol.

[0014] In this embodiment, the HMD 100 and the smartphone 200 cooperate with each other, and the HMD 100 is installed. The smartphone 200 sends a command to start the installed application (hereinafter referred to as the application). and the application is executed by the HMD 100. In this embodiment, this state is called a linked state. In the following, when there is no need to distinguish between the HMD 100 and the smartphone 200, they will be referred to as information terminals. Call.

[0015] In the first embodiment, whether the HMD 100 and the smartphone 200 are in a linked state is determined by the HMD 1 The decision is made by the 00 side. Whether or not there is a linkage state is determined by the placement of both sides. Specifically, As shown in FIG. 2(a), the display 131 (131a, 131b) of the HMD 100 The image displayed on the display 231 of the smartphone 200 is within the predetermined linked visual field 101a. If the app icon 420 (421, 422, 423) exists, it is determined that the device is connected. do.

[0016] In FIG. 2(a), application A (application A) is in the linked field of view 101a. The icon 421 of A) appears to be selected. The icon 420 of the application is displayed on the HMD 100 and the smartphone within the linked visual field 101a. The arrangement state with 200 is called a predetermined arrangement state.

[0017] When the HMD 100 and the smartphone 200 are in a predetermined arrangement state, it is determined that they are in a linked state. The user 501 is displayed on the smartphone 200 through the display 131 of the HMD 100. When the icon 421 is selected by touching it or the like (FIG. 2(a)), for example, the smartphone 20 On the HMD 100 side, an instruction to start the application corresponding to the selected icon 421 is given. The app is executed on the HMD 100 (Fig. 2(b)). 4 shows a state in which the application is executed and an icon 411 of the application is displayed.

[0018] In this embodiment, when the user 501 touches the icon 421 of the application A in the linked state, When the smartphone 200 accepts the selection by the user 501, the smartphone 200 and the HM D100, and touch information by the user 501 is transmitted from the smartphone 200 to H Sent to MD100.

[0019] When the HMD 100 receives touch information on the icon 421, it treats it as a start instruction. and launches app A in the HMD100.

[0020] As shown in FIG. 3(a), the smartphone 200 is located within the linked visual field 101a of the HMD 100. The arrangement state in which the icon 420 is not displayed on the display 231 is called a non-predetermined arrangement. This is called the position state.

[0021] In a non-predetermined arrangement state, the user 501 performs an action such as touching the icon 421 of the application A. Then, when the smartphone 200 accepts the selection of app A, the smartphone 200 continues to use the app. Start A.

[0022] In this way, in this embodiment, even if the HMD 100 is worn, the application can be displayed on the smartphone 200. If the user wishes to execute the operation, the icon 421 should be moved out of the range of the linked visual field 101a.

[0023] In this embodiment, the HMD 100 and the smartphone 200 are used for playing music and images, documents, etc. Each app has the same or corresponding app used to perform a specific task, such as creating The HMD 100 and the smartphone 200 are connected via wireless communication or wired communication between the devices. Communication allows information to be sent and received.

[0024] [Hardware configuration] Hereinafter, the HMD 10 constituting the information processing system 500 of this embodiment that realizes the above processing will be described. We will explain the hardware configuration for each of the smartphones.

[0025] 4 and 5 are block diagrams showing configuration examples of the HMD 100 and the smartphone 200, respectively. In these figures, the same components as those shown in FIGS. 1 to 3 are denoted by the same reference numerals.

[0026] FIG. 4 is a hardware configuration diagram of the HMD 100 of this embodiment. As shown in this diagram, The HMD 100 of this embodiment includes a main processor 111, a system bus 112, and a memory. device 110, an operation reception device 120, an image processing device 130, and a voice processing device 140, A sensor 150, a communication device 160, an expansion interface (I / F) unit 105, and a timer 106 and a vibration generator 107.

[0027] The main processor 111 is a main processor that controls the entire HMD 100 according to a predetermined program. The main processor 111 is a central processing unit (CPU). This is realized by a main processor (MPU) or a microprocessor unit (MPU). The processor 111 executes an operating system (Opera) stored in the storage device 110. Operating System (OS) and various operation control applications By executing this, the operation control process for the entire HMD 100 is performed. The main processor 111 controls the startup of various applications. The processing is performed according to the clock signal measured and output by the clock 106.

[0028] The system bus 112 transmits data between the main processor 111 and each component in the HMD 100. It is a data communication path for sending and receiving data.

[0029] The storage device 110 includes a RAM 103 and a ROM 104. Operating systems and programs such as music, image, document, and other various operation control applications. It also stores the base RAM required for basic operations such as the operating system. It stores information data such as data and file data launched by various applications. For example, when the music application installed on the HMD100 is launched and music file data is selected, When the music file is selected, the HMD 100 plays the music file data you selected. You can listen to it.

[0030] RAM 103 is used for basic operation programs and other application programs. The RAM 103 is a program area. RAM 103 is a temporary storage area that temporarily stores data as needed. It may be integrated with the processor 111.

[0031] The ROM 104 stores various operation setting values ​​of the HMD 100, information about the user of the HMD 100, etc. The ROM 104 stores still image data and moving image data captured by the HMD 100. The HMD 100 may also receive the information from an application server via the Internet. By downloading new application programs, you can expand the functionality. At this time, the downloaded new application program is The main processor 111 reads the new application data stored in the ROM 104. By loading and executing the application program in RAM 103, the HMD 100 , and can realize a variety of functions.

[0032] The ROM 104 stores the information even when the HMD 100 is not powered. Anything that can hold information is sufficient, and data can be read but not written. The ROM 104 is not limited to a mask ROM, which does not come with a built-in memory. For example, a flash ROM or a SS ROM can be used. D (Solid State Drive), HDD (Hard Disc Drive) ) and other devices are used.

[0033] The operation reception device 120 receives input of an operation instruction for the HMD 100. In this embodiment, the touch panel 120 includes an operation key 121 and a touch sensor 122. The operation key 121 is connected to a power The touch sensor 122 is a touch pad. It should be noted that the HMD 100 of this embodiment does not necessarily have to be able to perform all of these operations. The operation reception device 120 may not be provided. The operation of the HMD 100 may be received via a separate information processing terminal device connected to the HMD 100 .

[0034] The operation reception device 120 is located in a position and form within the HMD 100 that makes it easy for the user to perform input operations. It may be separated from the main body of the HMD 100 and connected by wire or wirelessly. Furthermore, it may be something that utilizes the line of sight of the user 501. For example, an input operation screen is displayed on the display 131, and the input detected by the in-camera 134 is Input operation information is acquired based on the position on the input operation screen to which the user's line of sight is directed. Even if you display a pointer on the input operation screen and operate the pointer to capture input operation information, Also, the user 501 may utter a voice indicating an input operation, and the voice may be collected by a microphone 143, which will be described later. The input operation information may be acquired by

[0035] The image processing device 130 is an image (video) processor and has a display 131 and , an outer camera 133 and an inner camera 134.

[0036] The display 131 is a display device (display) such as a liquid crystal panel, The image data processed by the image signal processing unit is presented to the user of the HMD 100. The display 131 includes a left-eye display 131a and a right-eye display 131b. The display 131 may be an optically transmissive display or a video transmissive display. It is also possible.

[0037] The optically transparent display displays playback information from the running application and notification information to the user 501. A projection unit that projects various information and a transparent screen that displays the projected information in front of the eyes. The video transmission type display is equipped with a half mirror. The device is equipped with a liquid crystal panel that displays the real-space object in front of the user along with various information. The display 131 allows the user 501 to see the image in front of them in their field of view as well as the launching action. You can also view playback information for music, images, documents, etc.

[0038] In this embodiment, the display 131 displays the execution status of the application and the like.

[0039] The image signal processing unit receives the image data from the outer camera 133 and the inner camera 134. It is an image (video) signal processor that processes images. The object created by the main processor 111 or the like is superimposed on the input image to display the image. The image signal processing unit is realized by the main processor 111. Alternatively, a processor dedicated to images may be provided in addition to the main processor 111, thereby It may be realized.

[0040] The outer camera 133 captures an image of the surroundings of the HMD 100. , which is installed in front of the HMD 100 and captures the forward field of view. The system uses a half mirror or other device to project various necessary information onto the real-world object in front of the user. The optical transmission type looks directly at the body, and the outer camera 133 captures the real-world object in front of the eyes and each A video transmission type that displays the information of the species on a display 131 such as a liquid crystal panel, and However, both cameras capture the visual field in front of the user's eyes using the outer camera 133. It is something that is captured.

[0041] The in-camera 134 captures an image of a different area from the out-camera 133. For example, In this embodiment, the in-camera 134 performs a gaze determination process. Together with this processing unit, it functions as a gaze detection sensor.

[0042] The gaze detection sensor includes a right eye gaze detection unit and a left eye gaze detection unit, The process of detecting the gaze is generally called eye tracking processing. For example, in the method using corneal reflection, infrared A line LED (Light Emitting Diode) is irradiated onto the face and an infrared camera is used to The position of the reflected light from the infrared LED on the cornea (corneal reflection) is used as the reference point. Detecting the gaze based on the position of the pupil relative to the position of the membrane reflection. In this case, an infrared camera, Equipped with an infrared LED.

[0043] The audio processing device 140 is an audio processor that processes audio. and a microphone 143.

[0044] The speaker 141 outputs the audio signal processed by the audio signal processing unit to the outside. The processing unit is an audio signal processor. etc. are used.

[0045] The microphone 143 converts the user's voice or the like into audio data and inputs it. The device may also include a microphone for picking up external sounds, and a speech microphone for picking up external sounds, respectively. The voice of the user 501 is collected. Headphones may be connected. The phone is worn on the ear of the user 501 and listens to the voices being spoken to the user 501 .

[0046] The sensor 150 is a group of sensors for detecting the state of the HMD 100. a GPS (Global Positioning System) receiving unit 151; A gyro sensor 152, a geomagnetic sensor 153, an acceleration sensor 154, and a depth sensor 1 These sensors are used to measure the position, movement, tilt, direction, etc. of the HMD 100. The depth sensor 155 also acquires distance information from the HMD 100 to the object. It should be noted that other sensors may also be provided.

[0047] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time. The gyro sensor 152 can detect movements, vibrations, shocks, etc. This sensor detects the angular velocity in the rotation direction and can capture the vertical, horizontal and diagonal posture. The acceleration sensor 154 and the gyro sensor 152 mounted on the HMD 100 are used. The movement of the HMD 100 can be detected by using the sensor.

[0048] The geomagnetic sensor 153 is a sensor that detects the magnetic force of the earth, and is It detects the direction of the earth's magnetism in the vertical direction as well as the forward / backward and left / right directions. By using the axis type, the HMD100 can detect changes in the earth's magnetic field in response to the movement of the HMD100. It is also possible to detect the movement of the smartphone 2 through the HMD 100. Alternatively, it may be possible to detect and determine whether the icon 420 on the display 231 of 00 is visible.

[0049] The depth sensor 155 is a sensor that can capture the shape of an object such as a person or an object in three dimensions. It is a device that irradiates an object with infrared or other laser light, measures the scattered light that bounces back, and LiDAR (Light Detection and Ranging) analyzes and detects the distance to an object and the state of that object. Detection and Ranging), and the reflection of pulsed light irradiated on the subject. TOF (Time Of Flight) sensor measures distance by measuring time for each pixel, It emits millimeter-wave radio waves, captures the reflected waves, and measures the distance to the object that is reflecting them and the object that is the object. The depth sensor 155 detects the state of the smartphone. Measure the distance to 200.

[0050] The communication device 160 communicates with nearby devices via short-range wireless communication, wireless LAN, or base station communication. A communication interface that communicates wirelessly with other information terminals, such as the smartphone 200 and the SW300 (described later). For wireless communication, the icon on the smartphone 200 is transmitted via a transmitting and receiving antenna. information about the device, the app to be launched, and the file data to be played by the launched app. Send and receive the following:

[0051] The communication device 160 is a LAN (Local Area Network) communication interface. A base (I / F) 161, a telephone network communication I / F 162, and a BT (Bluetooth (registered) The LAN communication I / F 161 is a communication I / F for the Internet. It connects to a wireless communication access point via wireless communication to send and receive data. The communication I / F 162 communicates with the base station of the mobile telephone network via wireless communication. The BT communication I / F 163 communicates with external devices according to the Bluetooth standard. This is an interface for communicating with external devices. The I / F 162 and the BT communication I / F 163 are respectively equipped with an encoding circuit, a decoding circuit, an antenna, etc. The communication device 160 may further include an infrared communication I / F or the like.

[0052] In addition to Bluetooth (registered trademark), IrDA is also available as a short-range wireless communication technology. (Infrared Data Association, registered trademark), Zigbee( Registered trademark), HomeRF (Home Radio Frequency, registered trademark), etc. Alternatively, it may be performed using a wireless LAN such as Wi-Fi (registered trademark). In addition, W-CDMA (Wideband Code Division Multiple Access) may be used for base station communication. Division Multiple Access) and GSM (Glo bal System for Mobile communications) Long-distance wireless communication can be used. The positional relationship and orientation between information terminals may be detected using the UWB (Ultra Wideband) technology. Therefore, the icon 420 on the display screen of the smartphone 200 can be seen through the HMD 100. The state of the battery may be detected and determined.

[0053] The communication device 160 may use other methods such as optical communication and sound wave communication as a means of wireless communication. In this case, instead of the transmitting and receiving antennas, a light emitting / receiving part and a sound wave emitting / receiving part may be used. A force / sound wave input unit is used.

[0054] The expansion interface unit 105 is an interface for expanding the functions of the HMD 100. In this embodiment, the group includes a charging terminal, a video / audio interface, a USB (Universal Serial Bus) It is equipped with a RS Serial Bus interface, memory interface, etc. The video / audio interface is used to input video / audio signals from external video / audio output devices. , output of video / audio signals to external video / audio input devices, etc. USB interface The USB interface connects to the USB device. The memory interface connects to the memory card or other memory media. It connects the body to send and receive data.

[0055] The vibration generator 107 generates vibrations under the control of the main processor 111. The vibration generator 107 converts notification information for the user 501 into vibration. The vibration generator 107 generates vibrations on the head of a user 501 who wears the HMD 100 closely. By generating the notification, the notification can be reliably conveyed to the user 501.

[0056] FIG. 5 is a hardware configuration diagram of the smartphone 200 according to this embodiment. As shown in this diagram, The smartphone 200 of this embodiment includes a main processor 201, a system bus 202, and a memory. device 210, an operation reception device 220, an image processing device 230, and a voice processing device 240, A sensor 250, a communication device 260, an expansion interface (I / F) unit 205, and a timer 206 and a vibration generator 207.

[0057] The details of each device are basically the same as those of the HMD100. The operation reception device 220 further includes a touch panel 223. The touch panel 223 is provided with a touch panel 123 that is placed on top of the display 231. In addition, instructions are input via a keyboard or the like connected to the expansion interface unit 205. That's fine.

[0058] The display 231 is configured with a liquid crystal panel or the like, and displays the remaining battery capacity, various accessories, and the like. The display 231 also displays notification information to the user 501, such as alarms, time, etc. The operation reception device 220 displays the icon of the application to be started and the execution status of the application. For example, it is a touchpad-type input method such as a capacitance type, which is input by a finger or a touch pen. The touch panel 501 detects an approach or contact operation (touch operation) as an operation input. Set and input the information you want to input, or use the app or file icons on the display 231. You can easily select and specify the icon by touch operation.

[0059] [Function Block] Next, the functional configurations of the HMD 100 and the smartphone 200 of this embodiment will be described. 6(a) and 6(b) are diagrams illustrating the HMD 100 and the smartphone of this embodiment, respectively. FIG. 2 is a functional block diagram of functions related to this embodiment of the E200.

[0060] As shown in FIG. 6(a), the HMD 100 includes a main control unit 170, a reception unit 171, and a transmission / reception unit. a signal receiving unit 172, a link determining unit 173, an application executing unit 174, an output unit 175, and an application The application program storage unit 176 and the icon data storage unit 177 are provided.

[0061] The main control unit 170 controls the operation of each unit of the HMD 100 .

[0062] The reception unit 171 receives instructions from the user 501. In this embodiment, This is realized by the processor 111 operating the operation reception device 120. For example, it accepts instructions to start an application.

[0063] The transmitting / receiving unit 172 transmits and receives data to and from external devices. This is realized by the processor 111 operating the communication device 160. Data is sent and received to and from the smartphone 200 via the receiving unit 172.

[0064] The cooperation determination unit 173 realizes cooperation processing with the smartphone 200. Whether the smartphone 200 is in a linked state in which it can operate in a linked operation mode described later is determined. In this embodiment, as shown in FIG. 2(a), the display 131 displays the linked visual field 1 01a, and analyzes the image captured by the outer camera 133 and displays it. Based on the displayed content, it is determined whether or not the predetermined arrangement state is achieved.

[0065] The associated visual field 101a is formed when the user 501 looks at the opening (lens part) of the HMD 100 or The display 131 is set to a part of the field of view that can be seen through the display area. It should be noted that the user 501 may change the device configuration, setting values, etc. The entire visual field range that can be reached may be set as the linked visual field 101a.

[0066] As shown in FIG. 2( a ), the cooperation determination unit 173 displays the image of the smartphone 200 on the captured image. If the icon 420 of the predetermined application on the screen is displayed, it is determined that the device is in the predetermined arrangement state. The predetermined application is, for example, an application that has received a notification from the smartphone 200. In this way, the smartphone 200 receives the fact that the user 501 has selected a predetermined icon 420. When the icon 420 is selected, information for identifying the application associated with the icon 420 (application identification information ) is sent to the HMD 100. The HMD 100 receives the application-specific information and The icon 420 associated with the application identified by the application identification information is identified.

[0067] When the predetermined arrangement state is reached, the cooperation determination unit 173 determines whether the application that received the notification from the smartphone 200 This program determines whether it is installed on itself. If the connection is established, the connection determination unit 173 of this embodiment determines that the connection is established. The icon is in a predetermined arrangement state and corresponds to the icon operated by the user 501. If the program is installed on the device, it is determined to be in a linked state.

[0068] When the cooperation determination unit 173 determines that the communication device is in a cooperation state, the cooperation determination unit 173 outputs information indicating that the communication device is in a cooperation state, i.e., That is, the connection status signal is transmitted to the smartphone 200. On the other hand, if the connection status is not determined to be established, A non-linked state signal indicating that the device is in a non-linked state is transmitted to the smartphone 200. The state signal does not need to be transmitted.

[0069] The HMD100 displays 420 icons of applications installed on the device. The shape data is stored in advance as icon data in the icon data storage unit 177. The data storage unit 177 stores shape information of the icon 420 in association with the application specifying information. The link determination unit 173 refers to the icon data storage unit 177 and stores the received icon. It is determined whether or not shape data associated with the pre-specific information is stored. If so, the shape data is extracted and the shape data is displayed in the area corresponding to the linked visual field 101a. It is then determined whether there is an image area that matches the data.

[0070] The cooperation determination unit 173 stores in the icon data storage unit 177 an icon corresponding to the received application identification information. If the attached shape data is not stored, it is determined to be in an unlinked state. Even if the data is stored, the extracted shape is stored in the area corresponding to the linked visual field 101a of the captured image. If there is no image area that matches the shape data, it is determined to be in an unlinked state. If there is an image area that is stored and matches the extracted shape data, it is determined to be in a linked state. do.

[0071] The captured image contains a shape corresponding to the display of the smartphone 200 and the shape of the icon 420. The determination of whether or not there is a state is made by pattern matching or the like.

[0072] When the reception unit 171 receives an instruction to start an application, the application execution unit 174 starts the application. The application execution unit executes the application program. It is stored in the application program storage unit 176.

[0073] The output unit 175 outputs the execution result of the executed application. The information is displayed on a display 131 and output as audio from a speaker 141.

[0074] The application program storage unit 176 stores application programs. can be.

[0075] The icon data storage unit 177 stores the icon data for receiving an instruction to execute each application program. The shape data of the icon 420 is stored as icon data. The shape data is stored in association with the application. The data is not limited to images, but may be text data, etc. The link determination unit 173 determines whether the app is linked by the icon design. If you cannot determine the application, you can identify the corresponding app using the accompanying text data. stomach.

[0076] Each of the above functions is implemented by the main processor 111 executing a program stored in the ROM 114. This is realized by loading it into AM113 and running it. The RAM memory section 176 and the icon data memory section 177 are constructed in the storage device 110 .

[0077] Next, the functions of the smartphone 200 of this embodiment will be described. It is a Noh block.

[0078] As shown in the figure, the smartphone 200 of this embodiment includes a main control unit 270, a reception unit 271, and , a transmitting / receiving unit 272, a cooperation control unit 273, an application execution unit 274, an output unit 275, and It includes an icon data storage unit 277 and an application program storage unit 276 .

[0079] The main control unit 270 controls the operation of each unit of the smartphone 200.

[0080] The reception unit 271 receives instructions from the user 501. In this embodiment, This is realized by the processor 211 operating the operation reception device 220. For example, it accepts various operational instructions such as instructions to launch an application.

[0081] The transmitting / receiving unit 272 transmits and receives data to and from external devices. This is realized by the processor 211 operating the communication device 260. Data is transmitted and received to and from the HMD 100 via the receiving unit 272 .

[0082] The cooperation control unit 273 determines the operation mode of the smartphone 200 and controls the smartphone in the determined operation mode. In this embodiment, the smartphone 200 alone operates as an application. and a linked operation mode that runs the app in conjunction with the HMD100. There are two types:

[0083] The cooperative operation mode is an operation mode that is executed when the devices are in a cooperative state. determines the execution device (execution terminal) that runs the app when in collaborative operation mode. However, in this embodiment, in the cooperative operation mode, the HMD 100 is determined as the execution device. Determine.

[0084] That is, in the cooperative operation mode, the smartphone 200 receives the icon 420 from the user 501. When an instruction to launch an app is received by touch, etc., the instruction is sent to the linked HMD. In this embodiment, the application is not executed on the own device.

[0085] In this embodiment, when the cooperation control unit 273 receives a cooperation status signal from the HMD 100, On the other hand, if a non-cooperative state signal is received, In this case, operate in standalone operation mode.

[0086] In the standalone operation mode, when an instruction to start an application is received from the user 501, the application starts on the device itself. This is the operating mode in which the app is launched and executed.

[0087] The application execution unit 274 is a second application execution unit that executes an application on the smartphone 200. In this embodiment, the reception unit 171 receives an instruction to start an application and executes a linkage system. If the control unit 273 determines that the application is in the standalone operation mode, the application program to be executed is executed. is stored in the application program storage unit 276.

[0088] The output unit 275 outputs the execution result of the application executed by the application execution unit 274. In this embodiment, the execution results are output to, for example, a display 231 and a speaker 241.

[0089] The application program storage unit 276 stores application programs. The icon data storage unit 277 stores the icon data for receiving an instruction to execute each application program. The icon is stored in association with the application. .

[0090] Each of the above functions is implemented by the main processor 211 executing a program stored in the ROM 214. This is realized by loading it into AM213 and running it. The RAM memory section 276 and the icon data memory section 277 are constructed in the memory device 210 .

[0091] [Processing flow] Next, a flow of application cooperation processing in the information processing system 500 of this embodiment will be described. In this embodiment, the HMD 100 and the smartphone 200 cooperate with each other to start an application of the HMD 100. FIG. 7 is a diagram illustrating the flow of application linkage processing.

[0092] This process is performed when the HMD 100 and the smartphone 200 are in a state where they can perform short-distance communication, for example. In the following description of the processing flow, the main control unit 17 of the HMD 100 0 and the main control unit 270 of the smartphone 200 are respectively It is written as follows.

[0093] The smartphone 200 and the HMD 100 first establish communication with each other (S1101, S120 In this embodiment, each of the transmission / reception units 172 and 272 is provided with, for example, a Bluetooth communication I / F 1 Communication is established by sending and receiving data via 63 and 263.

[0094] The smartphone 200 has four icons of apps that can be launched on the smartphone 200 on the display 231. 20 is displayed. The smartphone 200 receives a selection instruction from the user 501 (step S 1202). The reception unit 271 of the smartphone 200 receives the selected message displayed on the display 231. The touch operation on the icon 420 associated with the selected application is performed on the touch panel 223. Here, for example, it is assumed that application A is selected.

[0095] The transmitting / receiving unit 272 of the smartphone 200 is connected to the HMD 100 via the BT communication I / F 263. The selection information is notified (step S1203). , and app specific information that identifies the selected application (here, app A) is included. can be.

[0096] The cooperation determination unit 173 of the HMD 100 receives the application from the smartphone 200 via the transmission / reception unit 172. When the selection information of the HMD 100 is received (step S1102), the HMD 100 and the transmission source It is determined whether the terminal (smartphone 200) is in a linked state (step S1103). Here, whether the program of application A is installed in the own device and both terminals The arrangement state of the two is determined to be a predetermined arrangement state or not. Then, the selected program of application A is installed and the predetermined distribution is performed. If the device is in a location where it is connected, it is determined to be in a linked state. Otherwise, it is determined to be in a non-linked state. .

[0097] The HMD 100 determines whether it is in a linked state or not, that is, sends a link signal. The mobile terminal 100 returns either a link signal or a non-link signal to the smartphone 200 (step S1104).

[0098] When the cooperation control unit 273 of the smartphone 200 receives the determination result via the transmission / reception unit 272 ( Step S1204), and then the execution device is determined based on the result of the determination (Step S1205). ) In this case, if the result of the determination is that the device is in a linked state, the execution device will If the determination result is that the device is in a mobile state, the execution device is determined to be the smartphone 200.

[0099] The cooperation control unit 273 of the smartphone 200 transmits the decision result to the HMD 100 via the transmission / reception unit 272. (step S1206). The HMD 100 also receives the determination result (step Top S1105).

[0100] After that, depending on the determination result, the smartphone 200 and the HMD 100 each execute the application A. Execute.

[0101] When the smartphone 200 is determined as the execution device, the application execution unit 274 of the smartphone 200 (Step S1207), and then start up the application A in the smartphone 200 (Step S1208). The application A is executed (steps S1209 and S1210) until the application A is terminated. For example, if app A is a music playback app, it will play the playback information to the end. If an end instruction is received via the receiving unit 271, the process is terminated. It may be terminated.

[0102] On the other hand, if the execution device is not determined to be the smartphone 200, the process is terminated. do.

[0103] Similarly, the application execution unit 174 of the HMD 100 determines that the execution device is the HMD 100. If the application is received (step S1106), the application A in the HMD 100 is started (step S1 107), and executes application A until application A is terminated (steps S1108, S110 9) The process ends. Note that an instruction to end application A is usually given on the HMD 100 side.

[0104] On the other hand, if the execution device is not determined to be the HMD 100, the process ends. do.

[0105] When application A is executed on the HMD 100, for example, application A may store music, images, documents, etc. If it is an app to be played, start and run app A on the HMD100 and display 131, a speaker 141, etc. are used to play back information such as music, images, documents, etc. If an operation to instruct the end of playback is performed via the operation receiving device 120, the process ends. Alternatively, the process may be terminated when the playback information has been played back to the end.

[0106] When running app A on the HMD 100, the instruction to end app A is sent to the smartphone 200. For example, the smartphone 200 may receive the selected The smartphone 200 continues to display the icon 420. Then, the smartphone 200 notifies the user of the result of the determination in S1206. After that, if the icon 420 is touched, it is determined that an instruction to end the process has been received. and sends an end instruction to the HMD 100.

[0107] As described above, the information processing system 500 of this embodiment is a system in which a plurality of terminals cooperate with each other. The information processing system 500 executes the application that each terminal has in common. The system includes a first information terminal (HMD 100) and a second information terminal (smartphone 200) as terminals. The smartphone 200 includes a reception unit 271 that receives an instruction to start an application, and a stand-alone operation mode and and a cooperation control unit 273 that operates the smartphone 200 in either the cooperation operation mode or the cooperation operation mode. In addition, the HMD 100 is in a cooperative operation mode in which the smartphone 200 is operable in the cooperative operation mode. A cooperation determination unit 173 that determines whether the application is in the cooperation operation mode or not, and an execution unit 174 that executes the application in the cooperation operation mode. The output unit 175 outputs the result. In the linked operation mode, the app is run on both the HMD100 and the smartphone. 200. The cooperation control unit 27 of the smartphone 200 3, when the cooperation determination unit 173 of the HMD 100 determines that the HMD 100 is in a cooperation state, the HMD 100 operates in a cooperation operation mode. Make it.

[0108] The cooperative state in which the smartphone 200 can operate in the cooperative operation mode is the state in which the smartphone 200 can operate in the cooperative operation mode with the HMD 100. That is, the reception unit 271 of the smartphone 200 is determined by the arrangement of the smartphone 200. The display 231 (second display) of the host 200 displays an icon for receiving a start-up instruction. The HMD 100 displays the icon 420 and accepts a start instruction via the icon 420. The cooperation determination unit 173 determines whether or not the predetermined cooperation field 101 of the display 131 of the HMD 100 is in a predetermined cooperation field 101. If the icon 420 displayed on the display 231 is included in a, the device is in a linked state. It is determined that

[0109] In this manner, in this embodiment, the first information terminal exemplified by the HMD 100 and the smartphone 200 If the same app or a corresponding app is installed and can be launched on the second information terminal, An application displayed on a display 231 of a second information terminal with good operability as exemplified by E200. When the icon 420 of the first information terminal is touched, the arrangement state of the first information terminal and the second information terminal is The information terminal that launches the app is automatically determined depending on the user's preferences.

[0110] Specifically, when launching an app in the HMD 100, the smartphone 200 is connected to the HMD 100. By making the layout easy for the user 501 to understand by looking at the icon, In the linked state, an easy access can be performed using the smartphone 200 with good operability. By touching the icon, you can launch the desired application on the HMD100. The app you want to launch on the HMD100 can be displayed on the screen of the smartphone 200, which is easy to view and operate. Therefore, according to this embodiment, the operability and usability of the HMD 100 can be improved. The ease of use can be significantly improved.

[0111] Furthermore, according to this embodiment, when starting an application on the HMD 100, a smart operation with good operability is performed. Using HMD 200. App icons on Smartphone 200 are visible through the field of view of HMD 100. In this state, touch the app icon on the smartphone 200. By simply connecting the device to the HMD100, the same app or a compatible app can be launched via communication. It is possible.

[0112] This allows the user to perform touch operations on the icon of a desired application from the HMD 100. Since the touch panel 200 is visible, the user 501 can visually recognize the touch operation, and therefore the user can reliably perform the application. can be started.

[0113] For example, if the linked visual field 101a is set to the center of the display range of the display 131, Furthermore, it is highly reliable and convenient for the user 501.

[0114] On the other hand, according to this embodiment, the application icon of the smartphone 200 is displayed through the field of view of the HMD 100. If the smartphone is placed outside the specified position where it cannot be seen, the smartphone 200 itself can be used even if the HMD 100 is worn. The application installed on the device will launch as it is. For example, if the device is within the field of view of user 501, In the area where the icon 420 is located but is not determined to be in the predetermined arrangement state (area outside the linked visual field 101a), By looking at and touching the screen, you can use the app on the smartphone 200 while wearing the HMD 100. Furthermore, the HMD100 can be turned on even when not being worn or when wearing it. Even when the HMD 100 and smartphone 200 are not connected, such as when the power is turned off, The application can be launched directly on the smartphone 200. According to this, the user 501 can easily determine the terminal on which to launch the desired application. .

[0115] According to this embodiment, simply changing the arrangement of the smartphone 200 and the HMD 100 allows The smartphone 200 and the HMD 100 are linked together to use the smartphone 200 as an operating device for the HMD 100. The smartphone 200 is operated in a cooperative operation mode using the smartphone 200 alone, or in a standalone operation mode using the smartphone 200 alone. Therefore, the burden on the user 501 is reduced and the system is highly flexible. Moreover, it is possible to provide an information processing system 500 that is easy to operate.

[0116] In addition to the smartphone 200, the second information terminal can also display icons of apps and files. Any information terminal that can display and select icons, such as a tablet or smartwatch, It can also be a keyboard, a PC, etc.

[0117] In the above explanation, we will use music and video applications as examples of applications to be executed. However, the present invention is not limited to these.

[0118] <<Second embodiment>> Next, a second embodiment of the present invention will be described. The smartphone 200 determines whether the device 100 and the smartphone 200 are in a linked state, and executes the execution device In this embodiment, the HMD 100 determines the execution device.

[0119] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0120] The hardware configurations of the HMD 100 and the smartphone 200 are the same as those in the first embodiment. The functional configuration is also the same. However, the cooperation determination unit 173 and the cooperation control unit 2 73 is handled differently.

[0121] The cooperation determination unit 173 of the HMD 100 of this embodiment is the same as in the first embodiment. In this embodiment, the device 100 determines whether the device 100 and the smartphone 200 are in a linked state or a non-linked state. ,Then, determine the execution device of app A.

[0122] Specifically, when the cooperation determination unit 173 determines that the execution device is in a cooperation state, the cooperation determination unit 173 sets the execution device to H. If it is determined that the device is MD100 and that it is in an unlinked state, it sets the execution device as smartphone 200. Make a decision.

[0123] When the execution device is determined to be the HMD 100, the cooperation determination unit 173 determines whether the smartphone 200 is the execution device. Then, a signal (operation mode instruction signal) indicating that the operation should be performed in the cooperative operation mode is transmitted. On the other hand, when the execution device is determined to be the smartphone 200, the smartphone 200 is set to the independent operation mode. The CPU 10 transmits an operation mode instruction signal which indicates that the CPU 10 is to be executed in the normal mode.

[0124] In addition, the cooperation control unit 273 of the smartphone 200 of this embodiment is the same as the first embodiment. 00 is operated in the determined operation mode. The operation mode is not determined by the operation mode instruction signal sent from the HMD 100. The device operates in the specified operating mode.

[0125] [Processing flow] The flow of application linkage processing in the information processing system 500 of this embodiment will be described below with reference to FIG. The same processes as those in the application cooperation process of the first embodiment are denoted by the same reference numerals, and The explanation will be omitted.

[0126] The cooperation determination unit 173 of the HMD 100 of this embodiment is the same as the first embodiment. It is determined whether or not the smartphone 200 is in a linked state (step S1103). The device determination unit 173 determines the device on which application A is to be executed (step S2101). If it is determined that the device is in a linked state, it determines that the HMD 100 is the execution device and determines that the device is in a non-linked state. In this case, the execution device is determined to be the smartphone 200.

[0127] Then, the cooperation determination unit 173 transmits the decision result together with the determination result to the smartphone 200. (Step S2102). That is, an operation mode determination signal is transmitted to the smartphone 200. Thereafter, the process of step S1106 is executed according to the determination result.

[0128] When the cooperation control unit 273 on the smartphone 200 side receives the determination result (step S2201), According to the operation mode, the processing from step S1207 onwards is executed.

[0129] As described above, in the information processing system 500 of this embodiment, the cooperation of the HMD 100 When the determination unit 173 determines that the smartphone 200 is in a linked state, the determination unit 173 instructs the smartphone 200 to operate in a linked operation mode. Instruct them to do so.

[0130] As described above, according to the information processing system 500 of this embodiment, similarly to the first embodiment, The HMD 100 and the smartphone 200 can be smoothly linked without increasing the burden on the user 501. This provides an easy-to-use system that can be easily implemented.

[0131] <<Third Embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, whether or not the device is in a linked state and The smartphone 200 then determines the execution device.

[0132] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0133] The hardware configurations of the HMD 100 and the smartphone 200 are the same as those in the first embodiment. The functional configuration is also the same. However, the cooperation determination unit 173 and the cooperation control unit 2 73 is handled differently.

[0134] The cooperation determination unit 173 of the HMD 100 of this embodiment selects the application A from the smartphone 200. When the device receives application A selection information, which means that application A has been installed, the device The smartphone 200 determines whether the device is connected or disconnected, and transmits the determination result to the smartphone 200.

[0135] In addition, the cooperation control unit 273 of the smartphone 200 of this embodiment is configured to When the determination result of whether or not the HMD 100 is connected is received from the HMD 100, the device It is determined whether or not the device is in a linked state, and the execution device is determined.

[0136] The cooperation control unit 273 first checks whether the HMD 100 and the smartphone 200 are in a predetermined arrangement state. For example, as shown in FIG. 9, If the HMD 100 is present within the area 201a, it is determined that the smartphone 200 is in a predetermined placement state. The shape of the HMD 100 is stored as a pattern on the side, and is photographed by the in-camera 234. The captured image is analyzed, and if the pattern is present in the image corresponding to the photographing range 201a, The shape data of the HMD 100 is stored in the HMD 100 at the time of communication. It may also be used by sending it to the smartphone 200 from the side.

[0137] The cooperation control unit 273 determines that the HMD 100 and the smartphone 200 are in a predetermined arrangement state. In this case, the result of determining whether or not the application A has been installed from the HMD 100 is then referenced. If the result indicates that the device is installed, it is determined that the device is in a linked state. If the device is not in the specified position, or if the device is in the specified position, application A does not If it is not installed, it is determined to be in a non-linked state.

[0138] Then, the cooperation control unit 273 determines the execution device and notifies the result of the determination of the placement state. The result of the determination is transmitted to the HMD 100. For example, the cooperation control unit 273 If the device is determined to be unlinked, the HMD100 is selected as the execution device. The chair is decided to be Smartphone 200.

[0139] [Processing flow] The flow of application linkage processing in the information processing system 500 of this embodiment will be described below with reference to FIG. The same processes as those in the application cooperation process of the first embodiment are denoted by the same reference numerals, and The explanation of the degree is omitted.

[0140] The cooperation determination unit 173 of the HMD 100 of this embodiment selects the application A from the smartphone 200. When the application A selection information is received (step S1102), the application A is The smartphone 200 determines whether or not the smartphone A is installed, and transmits the determination result to the smartphone 200 ( Step S3101).

[0141] When the cooperation control unit 273 of the smartphone 200 receives the determination result from the HMD 100 (step The link status is determined (step S3201) and the link status is determined (step S3202). The determination result of whether or not both devices are in a predetermined arrangement state and the determination result received from the HMD 100 are also A decision is made based on the results (step S3202).

[0142] Then, the cooperation control unit 273 determines the execution device according to the result of the cooperation state determination ( In step S3203, the determination result of the cooperation status is transmitted to the HMD 100. do.

[0143] Thereafter, the processes from step S1207 onwards are executed depending on the determination result.

[0144] In addition, the HMD 100 receives the judgment result and the decision result from the smartphone 200 ( Step S3102), and depending on the determination result, the processes from step S1106 onwards are executed.

[0145] As described above, in the information processing system 500 of this embodiment, the smartphone 200 The control unit 273 determines whether the smartphone 200 is in a linked state in which it can operate in the linked operation mode. If it is determined that the smartphone 200 is in a linked state, the smartphone 200 is operated in a linked operation mode. do.

[0146] In this way, according to the information processing system of this embodiment, similarly to the first embodiment, Smooth collaboration between HMD100 and smartphone 200 without increasing the burden on 501 It is possible to provide an easy-to-use information processing system that can perform the above.

[0147] <<Fourth Embodiment>> Next, a fourth embodiment of the present invention will be described. In this embodiment, the execution of the selected application A is Even if the device is determined to be HMD100, the smartphone 200 is also running app A. This puts the application into a state where it can accept operations for the application A from the user 501.

[0148] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0149] The hardware configurations of the HMD 100 and the smartphone 200 are the same as those in the first embodiment. The functional configuration is also the same. However, the cooperation determination unit 173 and the cooperation control unit 2 73 is handled differently.

[0150] When it is determined that the smartphone 200 is in a linked state, the cooperation control unit 273 of the smartphone 200 Even if the device is determined to be HMD100, the smartphone 200 side also runs app A on the HMD Execute in sync with 100.

[0151] For example, as shown in FIG. 11(a), when the icon 421 of the application A is selected, The mobile control unit 273 transmits a start-up instruction to the HMD 100, and also on the smartphone 200 side, Then, as shown in FIG. 11(b), the display of the smartphone 200 The operation screen of the application A is displayed on the application 231. This shows the case where A is a music playback app.

[0152] In this embodiment, the cooperation control unit 273 performs the operation in a state where the execution device is determined to be the HMD 100. Then, via the display 231 of the smartphone 200, you can pause, skip, go back, stop, etc. When an operation for application A is accepted, a signal (operation signal) indicating the accepted operation is sent. , and transmit it to HMD100.

[0153] The cooperation determination unit 173 of the HMD 100 receives the status via the transmission / reception unit 172 while the application A is being executed. When an operation signal is received from the smartphone 200, the operation of the application A is controlled in accordance with the operation signal. do.

[0154] [Processing flow] The flow of application linkage processing in the information processing system 500 of this embodiment will be described below with reference to FIG. The same processes as those in the application cooperation process of the first embodiment are denoted by the same reference numerals, and The explanation of the degree is omitted.

[0155] In this embodiment, it is determined whether the HMD 100 and the smartphone 200 are in a linked state. The process of determining the execution device accordingly is carried out in any of the first to third embodiments. Here, this process is referred to as an execution device determination process.

[0156] In step S4400, the cooperation control unit 273 of the smartphone 200 determines an execution device. After the process is completed and the execution device is determined, the application execution unit 27 executes the process to start the application A. In response to this, the application execution unit 274 starts up the application A (step S 4201).

[0157] Then, depending on whether the execution device is a smartphone or not (step S1207), the process is performed. That is, when the execution device is determined to be the smartphone 200, the same as in the first to third embodiments. Then, the process from step S1209 onwards is carried out.

[0158] On the other hand, when the execution device is determined to be the HMD 100, the smartphone 200 also executes the application. The execution unit 274 executes the application A in synchronization with the HMD 100 (step S4202). Then, the reception unit 271 receives an operation for the application A via the display 231 or the like. When this is done (step S4203), the transmitting / receiving unit 272 transmits an operation signal corresponding to the operation to the HMD 10. 0. This continues until the end of application A. In FIG. 12, as an example, An example is given below when the termination operation of A is accepted.

[0159] Furthermore, on the HMD 100 side, even if the application A is being executed, the When receiving the operation signal from the smartphone 200, the application execution unit 174 executes the operation signal to the application A. For example, when an operation signal to instruct the end is received (as shown in FIG. 12), Application A is then terminated (step S4101), and the process is terminated.

[0160] As described above, in the information processing system 500 of this embodiment, in the collaborative operation mode, Even if you use the app on the HMD100, it will be synchronized and run on the smartphone 200. While the application is running, the smartphone 200 can accept operations for the application at any time. Cut.

[0161] That is, according to this embodiment, when the smartphone 200 and the HMD 100 are in a predetermined arrangement state, In this case, when you touch the icon of app A on the smartphone 200, app A will be displayed on the HMD 100. At this time, the smartphone 200 transitions to the operation screen of the application A, and the HMD 100 Accepts the operation of the running application A. For example, Touching the pause button on the operation screen resumes the playback of app A running on the HMD100. Life pauses.

[0162] In this way, according to the information processing system of this embodiment, similarly to the first embodiment, Smooth collaboration between HMD100 and smartphone 200 without increasing the burden on 501 Furthermore, this embodiment provides an information processing system that is easy to use. According to the company, even if an app is running on the HMD100, you can control the app on your smartphone. This can be done in 200, making it even easier to use and more convenient.

[0163] It is not necessary for the smartphone 200 to run the entire application. On the 0 side, only the I / F part is operated, and the operation signal for the received operation is sent to the HMD100 The HMD 100 may be configured to transmit the image data to the smartphone 2 and perform subsequent processing on the HMD 100 side. On the 00 side, operation buttons are displayed, operation input is accepted via the operation buttons, and operation signals are sent. The HMD100 app then plays the video according to the operation signal. This allows the resources of the smartphone 200 to be used efficiently and allows the smartphone 200 to have an easy-to-use screen. You can operate the HMD100 app with this.

[0164] <<Fifth Embodiment>> Next, a fifth embodiment of the present invention will be described. In this embodiment, the HMD 100 When the smartphone 200 is determined to be the execution device, the smartphone 200 transfers the data necessary for executing the application A to the H Before the application A is started in the MD 100, the application A is transmitted to the HMD 100. For example, this is an embodiment in which application A is an application for playing files, etc.

[0165] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0166] The hardware configurations of the HMD 100 and the smartphone 200 are the same as those in the first embodiment. The functional configuration is also the same. However, the cooperation determination unit 173 and the cooperation control unit 2 73 is handled differently.

[0167] In this embodiment, the cooperation control unit 273 of the smartphone 200 is configured to control the execution device H If it is determined to be MD100, that is, in the collaborative operation mode, execution of the selected application A The data required for this is sent to the HMD100.

[0168] After receiving the data transmitted from the smartphone 200, the cooperation determination unit 173 of the HMD 100 Start and run application A on the MD100.

[0169] [Processing flow] The flow of application linkage processing in the information processing system 500 of this embodiment will be described below with reference to FIG. The same processes as those in the application cooperation process of the first embodiment are denoted by the same reference numerals, and The explanation of the degree is omitted.

[0170] In this embodiment, it is determined whether the HMD 100 and the smartphone 200 are in a linked state. The process of determining the execution device accordingly is carried out in any of the first to third embodiments. Here, this process is referred to as an execution device determination process.

[0171] In step S4400, the cooperation control unit 273 of the smartphone 200 determines an execution device. Once the process is completed and the execution device is determined, the operation of the smartphone 200 is controlled accordingly. do.

[0172] That is, when the execution device is determined to be the HMD 100, the cooperation control unit 273 The data required for processing the application A is transferred to the HMD 100 by the reception unit 272 (step S5 201), the process ends. On the other hand, if the execution device is determined to be the smartphone 200, Execute the processing from step S1208 onwards.

[0173] Also, on the HMD 100 side, when the execution device is determined to be the HMD 100, the smartphone 2 When data is received from 00 (step S5101), the cooperation determination unit 173 The application A is started by the unit 174 (step S1107), and the process from step S1108 onward is performed. Execute.

[0174] As described above, in the information processing system 500 of this embodiment, When operating in the cooperative operation mode, the cooperative control unit 273 stores data necessary for executing the application. The data is transmitted to the HMD 100, and the HMD 100 uses the data received from the smartphone 200 to Run the pre-

[0175] In this way, according to the information processing system of this embodiment, similarly to the first embodiment, Smooth collaboration between HMD100 and smartphone 200 without increasing the burden on 501 It is possible to provide an easy-to-use information processing system that can perform the above.

[0176] In this embodiment, data required for executing the application is stored in the smartphone 200. It is desirable to reduce the storage capacity of the memory, etc., so there is no need to store it constantly in the HMD 100. Therefore, the HMD100 can run apps while keeping memory capacity low.

[0177] In addition, for example, if the application is a playback application for music, images, documents, etc., the The recorded data can be played back on the HMD100 using the playback app installed on the HMD100. In other words, this data can be reproduced on more familiar devices, which is more convenient.

[0178] In this embodiment, the application to be executed is first identified, and then its file data is transferred. However, the application to be executed is not limited to this. Only the data may be transferred. In this case, the HMD 100 side receives the file data. The application that processes the file data is then launched.

[0179] Also, if the same file is stored on the smartphone 200 and the HMD 100, The file in the HMD100 is not transferred, but only the file specification information is sent. It may be configured to play back on the HMD 100. This will eliminate the delay in playback due to data transfer. This can prevent this and reduce power consumption.

[0180] <<Sixth Embodiment>> Next, a sixth embodiment of the present invention will be described. In this embodiment, even in a linked state, The pre-processing is executed on the smartphone 200 side, and only the execution results are sent to the HMD 100.

[0181] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0182] The hardware configurations of the HMD 100 and the smartphone 200 are the same as those in the first embodiment. The functional configuration is also the same. However, the cooperation determination unit 173 and the cooperation control unit 2 73 is handled differently.

[0183] The cooperation control unit 273 of the smartphone 200 is in a cooperation state, and the execution device is the HMD 100. Even if it is determined that the application A is to be executed on the smartphone 200 side, the application A is executed on the HMD1 side. If 00 is the execution device, the execution result is not output to the device itself, but to the HMD100 Send to.

[0184] When the output device is determined to be the HMD 100, the cooperation determination unit 173 of the HMD 100 The execution result (output information) of the application A received via the transmitting / receiving unit 172 is output to the output unit 175. In addition, the output of the execution results requires a screen, processing, etc. specific to App A. In this case, application A is also launched on the HMD 100 to perform the processing.

[0185] [Processing flow] The flow of application linkage processing in the information processing system 500 of this embodiment will be described below with reference to FIG. The same processes as those in the application cooperation process of the first embodiment are denoted by the same reference numerals, and The explanation of the degree is omitted.

[0186] In this embodiment, it is determined whether the HMD 100 and the smartphone 200 are in a linked state. The output device to which app A will output is determined accordingly. The execution device determination process is a process for determining an execution device according to any one of the first to third embodiments. The output device is determined instead of the execution device in the same process as the execution device determination process.

[0187] In step S6400, the cooperation control unit 273 of the smartphone 200 determines an output device. When the process is completed and the output device is determined, the application execution unit 27 starts the application A. In response to this, the application execution unit 274 starts up the application A (step S 1208). Then, the cooperation control unit 273 causes the application execution unit 274 to execute the application A. (step S1209).

[0188] Then, when outputting, the output device determined in the output device determination process is Smartphone 2. It is determined whether the output device is HMD 100 or HMD 100 (step S6201). When the device is a smartphone 200, the cooperation control unit 273 outputs the The display 231, speaker 241, etc. of the device are output (step S6202). The process of steps S1209 to S6202 continues until application A is terminated, and then the process ends. do.

[0189] On the other hand, when the output device is determined to be the HMD 100, the cooperation control unit 273 The transmitting / receiving unit 272 transmits output information to the HMD 100 (step Step S6203).

[0190] On the HMD 100 side, if the output device is determined to be the smartphone 200, the process is continued as is. (step S6101), and if the HMD 100 is determined, the cooperation determination unit 173 The step S100 prepares for receiving output information by instructing the transmitting / receiving unit 172 to receive the output information (step S100). (Top S6102).

[0191] Then, when output information is received from the smartphone 200 (step S6103), the cooperation determination unit The communication unit 173 causes the output unit 175 to output the received output information (step S6104). The connection determination unit 173 continues the processes of S6103 and S6104 until the reception of the output information is completed. (Step S6105), and the process ends. For example, when an output information end signal is received, the specified time has elapsed since the output information could no longer be received. This may be determined by the passage of time, the loss of communication with the smartphone, etc.

[0192] As described above, in the information processing system 500 of this embodiment, the smartphone 200 When the smartphone 200 is operated in the cooperative operation mode, the control unit 273 executes the selected application. The execution result is sent to the HMD 100.

[0193] In this way, according to the information processing system of this embodiment, similarly to the first embodiment, Smooth collaboration between HMD100 and smartphone 200 without increasing the burden on 501 It is possible to provide an easy-to-use information processing system that can perform the above.

[0194] In particular, according to this embodiment, for example, when the smartphone 200 is viewed through the HMD 100, In the predetermined arrangement state, when the user 501 touches the music application icon 420, The software is executed on the smartphone 200, and only the audio playback information is sent to the HMD 100. It is possible to listen to the video and audio through the speakers (headphones). It is equipped with memory for extra information data such as large-capacity content data such as Raku, and the app itself. In other words, it is possible to reduce the amount of storage required while still allowing you to enjoy music as if it were an HMD100. It will be possible to operate as if the app was launched and played.

[0195] This reduces the storage capacity of the HMD100 and also reduces the required processing power. This allows the HMD 100 to be made smaller and lighter. According to the company, even if the HMD 100 is running, it can be operated from the smartphone 200, so it is easy to use. It's also convenient.

[0196] <<Seventh Embodiment>> Next, a seventh embodiment of the present invention will be described. As shown in 15, a first information terminal, a second information terminal, and a third information terminal are provided. It works in conjunction with information terminals to run apps.

[0197] In the following, in this embodiment, the first information terminal is a smart watch (SW300), and the second information terminal is a smart watch (SW300). An example will be described in which the information terminal is a smartphone 200 and the third information terminal is an HMD 100.

[0198] In this embodiment, the HMD 100 is in a position where the smartphone 200 and the SW 300 can cooperate with each other. Then, based on the result, the smartphone 200 executes the selected application. A can be executed on the device itself or on SW300, i.e., in the linked operation mode. The smartphone 200 determines whether to operate in the stand-alone operation mode or the stand-alone operation mode. When you do this, the selected app A is installed in SW300 in advance. Check that it is.

[0199] The configuration of each device in this embodiment will be described below. The hardware configuration is the same as that of the first embodiment, and the functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0200] Hardware configuration and functional blocks of the HMD 100 and smartphone 200 of this embodiment However, the cooperation determination unit 173 and the cooperation control unit 27 The processing of 3 is different.

[0201] The HMD 100 of this embodiment is in a position where it can be linked as a third information terminal. That is, the cooperation determination unit 173 of this embodiment only determines whether the smartphone 200 and , SW300 and SW301 are in a state where they can cooperate with each other.

[0202] Specifically, as shown in FIG. 16(a), within the linked visual field 101a of the display 131, , the screen on which the icon 420 of the smartphone 200 is displayed and the SW300 are both present. If the smartphone 200 is in the installed state, it is determined that the smartphone 200 and the SW 300 are in a state where they can cooperate with each other. The app program corresponding to icon 420 is installed on the SW300. In this case, the SW300 executes the selected application, as shown in FIG. 16(b).

[0203] The hardware configuration of the SW300 of this embodiment is shown in FIG. The SW 300 of the embodiment includes a main processor 311, a system bus 312, and a storage device. 310, an operation reception device 320, an image processing device 330, a voice processing device 340, and a sensor 350, a communication device 360, an extension interface (I / F) unit 305, and a timer 30 6 and a vibration generator 307.

[0204] The details of each device are basically the same as those of the HMD100. The operation reception device 320 further includes a touch panel 323. 323 includes a touch panel 323 arranged over the display 331.

[0205] [Function Block] Next, the functional blocks of SW300 are shown in FIG. 18. As shown in this figure, the main control unit 37 0, a reception unit 371, a transmission / reception unit 372, a cooperation control unit 373, and an application execution unit 374. , an output unit 375, an icon data storage unit 377, and an application program storage unit 376 and equipped.

[0206] Each functional block has the same function as the HMD100's configuration with the same name. The cooperation control unit 373 of the SW300 is the same as the cooperation control unit 273 of the smartphone 200. It is determined whether or not the device operates in a cooperative operation mode in which the application is executed in cooperation with the terminal. If it is determined that the cooperative operation mode is selected, the application execution unit 3 starts the cooperative operation mode in response to a start instruction from another information terminal. Launch the app on 74.

[0207] [Processing flow] The flow of application linking processing in the information processing system 500 of this embodiment will be described below. 9 shows the processing flow of the application linking process of this embodiment. The same reference numerals are used for the same components, and detailed explanations will be omitted.

[0208] First, the HMD 100 and the smartphone 200, and the SW 300 and the smartphone 200 are Communication is established (steps S1101, S1201, S1301). Each of the transceivers 172, 272, 372 performs data communication and establishes communication.

[0209] The smartphone 200 receives an instruction to select the application A (step S1202).

[0210] The smartphone 200 notifies the HMD 100 and the SW 300 of the selection information of the application A ( Step S1203).

[0211] The cooperation determination unit 173 of the HMD 100 determines whether the smartphone 200 and the SW 300 are in cooperation with each other by the above method. It is determined whether the arrangement is possible (step S7101), and the result of the determination is transmitted to the smartphone 20. 0 (step S7102). Note that communication is also performed between the HMD 100 and SW 300. The determination result may be transmitted to SW300.

[0212] In addition, the cooperation control unit 37 of the SW 300 that received the notification of the application A selection information from the smartphone 200 3 determines whether application A is installed on the device and reports the result to application A. The information is sent to the smartphone 200 (step S7303).

[0213] The placement status determination result was received from the HMD100, and application A information was received from the SW300 ( Step S7201) The cooperation control unit 273 of the smartphone 200 uses this information to The execution device that executes the application A is determined to be in a linked state with the SW300. In this embodiment, the arrangement state is such that cooperation is possible, and If you get information that app A is installed on SW300, you can set the execution device to S In other cases, the execution device is determined to be the smartphone 200.

[0214] The cooperation control unit 273 of the smartphone 200 notifies the HMD 100 and the SW 300 of the determination result. Upon receiving the determination result, the HMD 100 ends the process (step S7203).

[0215] After that, the smartphone 200 executes the processes from step S1207 onwards.

[0216] In addition, when the SW300 is determined to be the execution device (step S730 5) Application A in SW300 is started (step S7306), and application A is terminated. Application A is executed until the end of the process (steps S7307 and S7308).

[0217] On the other hand, if the execution device is not determined to be SW300, the process is terminated. .

[0218] As described above, the information processing system 500 of this embodiment includes a plurality of terminals, each of which is a HM. The HMD 100 includes a D100, a smartphone 200, and an SW 300. Then, the smartphone 2 determines whether it is in a linked state where it can operate in the linked operation mode. 00 operates in the linked mode when it is determined to be in linked status, and starts the selected application. The SW300 receives the start instruction and starts and runs the application. .

[0219] As described above, according to this embodiment, the application can be started in SW300 in a manner that is easy to operate. This can be done by a good smartphone 200. In addition, the cooperation between the two can be achieved by a third information terminal This is achieved by determining the relative positions of the two through the field of view of the HMD100. In this embodiment, the operability can be improved when executing an application. In this embodiment, a highly convenient information terminal cooperation technique can be realized without burdening the user 501. do.

[0220] Furthermore, according to the present embodiment, when starting up an application, the smartphone 200 with good operability is used, and H The smartphone 200 icon and SW300 are positioned so that they can be seen through the MD100's field of view. If there is one, touch the icon on the smartphone 200 in this state to display the location on the SW300. Therefore, the SW300 display and operation panel are simple. In other words, the SW300 configuration can be simplified.

[0221] In this embodiment, when the smartphone 200 is within the linked visual field 101a of the HMD 100, The SW300 is in a predetermined position and is within the linked visual field 101a of the HMD 100. In the case of a placement state where the HMD 100 is not set, the execution device is determined to be the HMD 100, as in the first embodiment. In this case, the smartphone 200 may transmit a start instruction to the HMD 100, and the Launch app A.

[0222] In this embodiment, the first information terminal is not limited to SW300. If the information terminal can be moved and the position can be easily adjusted to the optimum state, for example, a tablet, It may be a PC, a smartphone 200 other than the second information terminal, or the like.

[0223] In this embodiment, the SW300 is also used as a smartphone, as in the fourth embodiment. The app runs on the Home 200, synchronized with the SW300, and the operation is accepted by the Smartphone 200. Also, as in the fifth embodiment, the SW 300 receives the following from the smartphone 200 before executing the application: Furthermore, as in the sixth embodiment, the smartphone 200 may execute an application. Alternatively, the system may be configured to execute the program and receive and output only the execution results.

[0224] Transfer the file data stored in Smartphone 200 to SW300 and By playing data using the built-in file playback application, the SW300 can There is no need to store all the file data in one place, and it is a small, multi-functional watch like the SW300. This can reduce the data capacity, which is important for mobile information terminals.

[0225] <Variation 1> In the above embodiments, whether or not the two information terminals are in a predetermined arrangement state is determined by the two This is determined by whether the information terminals are in a predetermined positional relationship. The icon 420 on the display 231 of the smartphone 200 is present within the field of view 101a. However, the method of determination is not limited to this. do not have.

[0226] For example, ultra-wideband (UWB) wireless systems are used. The positional relationship and orientation between the information terminals are detected, and the smartphone 200 and HMD 100 are connected based on the results. It may be possible to identify and determine whether the smartphone 200 and the HMD 10 are in a predetermined arrangement state. The output information of various sensors 150 and 250 mounted on each device is used to calculate the Determines movement, detects the relative positions and orientations between devices, and determines whether they are in the specified arrangement It may also be used for

[0227] Furthermore, whether or not the device is in a predetermined arrangement state can be determined based on, for example, the distance between two information terminals. For example, the distance between the HMD 100 and the smartphone 200 may be used as a basis for the determination. If the distance between the user and the person is less than a predetermined distance threshold, the user is determined to be in a predetermined placement state.

[0228] The distance between them is, for example, a distance between the depth sensor of the HMD 100 in the first, second, and seventh embodiments. In the third embodiment, the depth is measured by the depth sensor 255 of the smartphone 200. do.

[0229] In particular, when determining whether or not the predetermined arrangement state is achieved based on the distance between the two, the processing is simple. That is, the user does not care whether the smartphone 200 is within the linked visual field 101a of the HMD 100. By touching the icon using a smartphone 200 located in the vicinity of a certain range, The desired application can be started on the HMD 100. In addition, the smartphone 200 can be kept in a constant state intentionally. By moving the device away from the smartphone 200 by more than this distance, the smartphone 200 can start the application. In addition, compared to the process of determining using the linked visual field 101a, This reduces the chance of accidentally removing something when you think you have.

[0230] In addition, the linked visual field 101a is not determined, and the direction of the user's line of sight is detected and displayed on the display 1. When the icon 420 of the smartphone 200 overlaps with a predetermined area including the intersection with the line of sight direction on 31 , may be in a predetermined arrangement state.

[0231] In this case, while wearing the HMD 100, the eye of the smartphone 200 is displayed on the display 131. Even if the cone 420 is visible, unless the user 501 is looking at it, the HMD 100 Therefore, it is possible to provide a device with higher operability.

[0232] Furthermore, the linked visual field 101a is set, and the icon 420 is placed within the range of the linked visual field 101a. If the user 501 is looking at the icon 420, it is determined that the icon is in a predetermined arrangement state. The device may be configured to:

[0233] <Variation 2> In the above embodiment, when it is determined that the devices are in a linked state, the devices are respectively provided with predetermined information. However, the method for determining the device on which the app is running is not limited to this. Not determined.

[0234] For example, after it is determined that the smartphone 200 is in a linked state, the icon 420 displayed on the smartphone 200 is The device on which the application is to be executed may be determined depending on the manner of the selection operation on the device. For example, If you single tap the icon 420, it will be 200 for your smartphone, and if you double tap, it will be H For example, start it on MD100.

[0235] Also, whether or not the device is in a linked state is determined by a different method, and only the execution device is viewed from the user 501's perspective. The line direction of the user 501 may be detected, and the line direction may be determined based on the line direction of the icon 420 or If the smartphone 200 is not on the display 231, the HMD 100 may prevent the app from starting.

[0236] This allows the user to take their eyes off the smartphone 200 while keeping the smartphone 200 within their field of vision. By operating the app, you can intentionally launch the app on the smartphone 200 instead of the HMD 100. can be done.

[0237] In addition, the execution device may be determined not only by the line of sight but also by whether the user 501 is opening or closing their eyes. In this case, the image captured by the in-camera 134 for a predetermined period is analyzed, and the It is possible to determine whether the eyes are closed intentionally for a certain specified period of time. For example, if you touch the icon 420 with your eyes closed for a certain period of time, Decide to launch the app on your Maho 200.

[0238] Also, if you intentionally close your eyes when you tap the icon and release your finger, the smartphone will You can also set it to start with 0. In this case, you can set it to start with the intended value while using the HMD100 all the time. This is useful when you want to explicitly start it on the smartphone 200.

[0239] Also, if one eye is closed, the smartphone 200 may be activated. It is possible to easily perform a desired action while looking at the icon 420 with one eye.

[0240] <Variation 3> In each of the above embodiments, data transmission and reception between the information terminals is performed by direct wireless communication between the terminals. For example, as shown in FIG. 20(a) and FIG. 20(b), the server 610 It may also be via.

[0241] Similarly, applications, files, and data are stored on the server 610 and can be accessed from the smartphone 200. The data may be transferred to each information terminal by communication as appropriate according to instructions from the smartphone 200. In response to these instructions, the application is started on the server 610 and playback information is determined according to each embodiment. The content may be transferred to a suitable information terminal and viewed there.

[0242] The server 610 can connect to each information terminal via communication to send and receive necessary information. For example, local servers, cloud servers, edge servers, internet services, etc. And the form doesn't matter.

[0243] In addition, in addition to transferring large-volume content data from the smartphone 200 to the HMD 100, , content data may be downloaded to the HMD 100 from a cloud network. In particular, image applications that handle high-definition images use a dramatically larger amount of data. Therefore, the capacity reduction effect of the HMD100 is large. In this case, 5G is used for data communication. 5th Generation: 5th generation mobile communication system) Local 5G and other high-speed, large-capacity The use of mass communication networks can further improve usability.

[0244] <Variation 4> Also, the icon 420 that the user 501 touches will be displayed on the display screen, and the information terminal on which the application is being used will be displayed. A small icon (hereinafter referred to as a terminal icon) indicates whether a program or file is executable. An example of an icon with a small icon is shown in FIG. 21(c) are shown. Here, as an example, a music playback application is assumed to be application A. Shows icon 430.

[0245] FIG. 21(a) shows an icon 430 in which the application is displayed on the HMD 100. This is an example of a case where a small icon 431 indicating that the program can be started is provided. ) indicates that the icon 430 is a small icon indicating that this app A can be moved using the SW300. This is an example in which a controller 432 is provided.

[0246] FIG. 21(c) shows an example of an application that can be activated only by the smartphone 200. The icon 440 of the spreadsheet application is shown. The small icon is not displayed.

[0247] As a result, when the user touches an icon on the smartphone 200, The small icons shown in Figures 21(a), 21(b), and 21(c) are displayed in the icons. While checking the status including whether the icon is present, touching this icon will start on which information terminal. This allows users to know in advance whether the function is possible, and makes it easy to improve usability.

[0248] The small icons explained in Fig. 21(a), Fig. 21(b), and Fig. 21(c) are displayed on the information terminal. Regardless of whether the interval is in the predetermined configuration state or not, it can be started when the interval is in the predetermined configuration state. This indicates an information terminal.

[0249] Furthermore, the cooperation control unit 273 of this embodiment receives the following from the HMD 100 or SW 300: When the information terminal receives information that the application A is installed, the small icon 431 is displayed as an icon Or, you can set up the HMD 100, SW 300, and smartphone 2 in advance. 00 to exchange information about whether the application program is installed. Alternatively, a small icon 431 may be displayed.

[0250] Other examples of small icons 431 are shown in Figs. 21(d) and 21(e). 31, when the icon 420 is touched, applications and files are launched between the linked information terminals. The information terminals that are connected may be displayed in a manner that allows them to be identified. Therefore, the display mode of the small icon 431 may be changed.

[0251] For example, when the smartphone 200 and the HMD 100 are in a linked state and the icon 420 is touched, When an application or file is launched on the HMD100, as shown in Figure 21(d), The small icon 432 with the light (shaded) is displayed. On the other hand, the smartphone 2 is not in a linked state. If the application starts with 00, it will be displayed without highlighting (shading) as shown in Figure 21(e). Display small icon 433.

[0252] In this way, by displaying the small icon 431, the user 501 can easily find out the information that can be activated. This allows the user 501 to easily know which information terminal to start. You can choose clearly with your will.

[0253] In addition to the display of the small icon 431, the method of identification and notification is to determine which information terminal the application is started on. The user 501 is notified via a speaker 141 (headphones) whether the operation is possible or not as audio information. That's fine.

[0254] In addition, the HMD100 determines which information terminals can launch the app based on the magnitude and type of vibration. Therefore, the user 501 may be notified of the identified vibration information by the vibration generator 107. This provides greater convenience to the user 501.

[0255] Whether or not to display small icons is determined by the initial setting for each app. This can be specified during pre-installation. Also, the user 501 can change the initial setting. Also, you can make display decisions for all apps on a device-by-device basis, not on an app-by-app basis. Even if the disconnections are all collectively controlled by the OS etc. and can be switched by the user 501's instruction, good.

[0256] <Variation 5> In the above embodiments and modifications, the icon touch detection method is performed using the smartphone 2. 00 operation reception device 120 detects the icon touch operation. For example, the outer camera 133 of the HMD 100 can be used to capture images on the smartphone 200. The touch operation may be detected by capturing a picture of the touch operation and analyzing the captured image. As a method for identifying the position of the HMD, the HMD 100 is directed to the smartphone 200. The coordinates on the display 231 of the smartphone 200 captured by the camera 133 are sent to the smartphone 20 0 may be used to identify the app at that coordinate and notify the HMD 100 of it.

[0257] <Variation 6> In addition, in the above-described embodiments, the HMD 100 and the smartphone 200 are given as examples of information terminals. However, any information terminal that can acquire location information between multiple information terminals is acceptable. In this case, the icon 420 is displayed on the smartphone and can be touched. By touching the icon while taking a photo of the smartphone 200A with the camera of another smartphone 200B, The app may be launched on another smartphone 200B.

[0258] In addition, the same application may be operated in cooperation between the information terminals. When using the same map app on the HMD100 and HMD100, if both devices are in a specified position, The smartphone 200 displays a detailed map, while the HMD 100 displays a simplified map. By linking these functions together, usability can be improved.

[0259] <Variation 7> In addition, when installing the same app or a corresponding app, install it on one of the information terminals. Once you have installed it, you can synchronize and install it on the other device. The same app or a corresponding app selected on one device is not available on the other device. In this case, you can download and install it from a server or other information terminal. In addition, the linking between information terminals (pairing, group setting, etc.) may be performed in advance. If communication is established with an information terminal that is not associated, association is performed. If the user 501 indicates his / her intention to link the data, Linking may be performed at this point.

[0260] The present invention is not limited to the above-described embodiment, but includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention. The present invention is not limited to an embodiment having all of the configurations described above. It is possible to replace a part of the configuration of an embodiment with the configuration of another embodiment, and it is also possible to replace a part of the configuration of an embodiment with the configuration of another embodiment. It is also possible to add the configuration of other embodiments to the configuration of this embodiment. It is possible to add, remove, or replace other configurations.

[0261] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in part or in whole by, for example, It may be realized by hardware such as by designing an integrated circuit. The processor interprets and executes the programs that realize the respective functions. It may be realized by software. Programs, tables, files, etc. that realize each function Information is stored in memory, hard disks, SSDs (Solid State Drives), etc. It can be placed on a recording device, or on a recording medium such as an IC card, SD card, or DVD. .

[0262] In addition, control lines and information lines are shown as they are considered necessary for explanation purposes, and do not necessarily represent all the lines in the product. This does not necessarily show all control and information lines. In reality, almost all components are interconnected. It can be said that this is the case. [Explanation of symbols]

[0263] 100: HMD, 101a: Coordinated vision, 103: RAM, 104: ROM, 105: Expanded Extension interface unit, 106: timer, 107: vibration generator, 110: storage device, 11 1: Main processor, 112: System bus, 113: RAM, 114: ROM, 12 0: operation reception device, 121: operation key, 122: touch sensor, 123: touch panel, 130: image processing device, 131: display, 131a: left-eye display, 131 b: right eye display, 133: outer camera, 134: inner camera, 140: audio processing processing device, 141: speaker, 143: microphone, 150: sensor, 151: receiving unit, 152 : Gyro sensor, 153: Geomagnetic sensor, 154: Acceleration sensor, 155: Depth sensor , 160: Communication device, 161: LAN communication I / F, 162: Telephone network communication I / F, 163: BT communication I / F, 170: main control unit, 171: reception unit, 172: transmission / reception unit, 173: cooperation Determination unit, 174: application execution unit, 175: output unit, 176: application program Storage unit, 177: icon data storage unit, 200: Smartphone, 200A: Smartphone, 200B: Smartphone, 201: Main processor, 201a: shooting range, 202: system bus, 205: expansion interface unit, 206: Timer, 207: vibration generator, 210: storage device, 211: main processor, 213 :RAM, 214:ROM, 220:Operation reception device, 223:Touch panel, 230:Image Image processing device, 231: display, 234: in-camera, 240: audio processing device, 24 1: speaker, 250: sensor, 255: depth sensor, 260: communication device, 263: BT Communication I / F, 270: main control unit, 271: reception unit, 272: transmission / reception unit, 273: cooperation control unit, 274: application execution unit, 275: output unit, 276: application program storage Unit 277: Icon data storage unit 300: Smart watch, 305: Expansion interface unit, 306: Timer, 307 : vibration generator, 310: storage device, 311: main processor, 312: system bus 320: Operation reception device, 323: Touch panel, 330: Image processing device, 331: Display Spray, 340: Audio processing device, 350: Sensor, 360: Communication device, 370: Main control unit, 371: reception unit, 372: transmission / reception unit, 373: cooperation control unit, 374: application execution unit, 375: Output section, 376: Application program storage section, 377: Icon data storage section, 411: Icon, 420: Icon, 421: Icon, 430: Icon, 431 :Small icon, 432:Small icon, 433:Small icon, 440:Icon, 500: Information processing system, 501: user, 610: server

Claims

1. An information processing terminal that cooperates with other external terminals to execute an application that is commonly provided to each terminal, The information processing terminal an output unit that outputs an execution result of the application executed in a cooperative operation mode, which is an operation mode in which the application is executed in the information processing terminal and the other external terminal in cooperation with each other; a link determination unit that determines whether the other external terminal is in a linked state in which it can operate in the linked operation mode, the linked state is determined by an arrangement of the information processing terminal and the other external terminal; the cooperation determination unit determines that the information processing terminal is in the cooperation state when an icon for accepting a start instruction displayed on the display of the other external terminal is included within a predetermined cooperation field of view of the display of the information processing terminal; An information processing terminal characterized by the above.

2. An information processing terminal that cooperates with other external terminals to execute an application that is commonly provided to each terminal, The information processing terminal a reception unit that receives an instruction to start the application; a cooperation control unit that operates the information processing terminal in either a standalone operation mode or a cooperative operation mode, the independent operation mode is an operation mode in which the application is executed independently by the information processing terminal; the cooperative operation mode is an operation mode in which the application is executed cooperatively between the other external terminal and the information processing terminal, The cooperation control unit determining whether the information processing terminal is in a linked state in which it can operate in the linked operation mode, and if it is determined that the information processing terminal is in the linked state, operating the information processing terminal in the linked operation mode; determining that the other external terminal is in the linked state when the other external terminal is within a predetermined range from the information processing terminal; An information processing terminal characterized by the above.

Images