Information Processing Systems

Abstract

To provide an information processing system that allows a user to easily start and execute an application on a desired device without increasing a burden on the user between a plurality of information terminals equipped with the same application or a corresponding application.SOLUTION: An information processing system includes a first information terminal (100) and a second information terminal (200). The second information terminal includes: a reception unit (271) that receives an instruction to start an application; and a cooperation control unit (273) that operates the second information terminal in either an independent operation mode or a cooperation operation mode. The first information terminal includes an output unit (175) that outputs an execution result of an application executed in the cooperation operation mode. The independent operation mode is an operation mode where the application is executed independently on the second information terminal, and the cooperation operation mode is an operation mode where an application is executed cooperatively on the first and second information terminals. A cooperation state where the second information terminal is operable in the cooperation operation mode is determined by arrangement of the first information terminal and the second information terminal.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for controlling the launch of applications (hereinafter referred to as apps) among a plurality of information terminals. [Background technology]

[0002] Because a head mounted display (HMD) has a small display area, it is sometimes used in conjunction with other information terminals. For example, Patent Document 1 discloses an application collaboration device including "a first management unit that manages device information including information used to connect to a collaboration destination device and information regarding data used by apps registered in the collaboration destination device; a second management unit that, in response to a collaboration request from a collaboration source device, refers to the device information and acquires collaboration candidates indicating collaboration destination devices and apps that can be collaborated with; and a collaboration control unit that starts the apps indicated in the collaboration candidates and determines the device to perform collaboration and the apps registered in the device that can be collaborated with as collaboration destinations (abstract excerpt)." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-142693 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology of Patent Document 1, operability is improved by selecting and specifying a desired application using another information terminal. However, with the technology of Patent Document 1, the application is launched on all information terminals on which the application is registered. This means that the application may be executed on an unintended information terminal, which may cause other applications on the information terminal to stop operating, thereby interfering with work.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a user-friendly information processing technology that allows an application to be easily launched and executed on a desired device, without increasing the burden on the user, across multiple information terminals equipped with the same application or a corresponding application. [Means for solving the problem]

[0006] The present invention is an information processing system in which a plurality of terminals cooperate to execute an application that is commonly provided to each terminal, the plurality of terminals including a first information terminal and a second information terminal. 、 and wherein the second information terminal receives an instruction to start the application. 、 The second information terminal is operated in either a standalone operation mode or a linked operation mode. And before the independent operation mode is an operation mode in which the application is executed independently on the second information terminal; the collaborative operation mode is an operation mode in which the first information terminal and the second information terminal execute the application in collaboration with each other, and the first information terminal outputs an execution result of the application executed in the collaborative operation mode; The linked state in which the second information terminal is operable in the linked operation mode is determined by the arrangement of the first information terminal and the second information terminal. the second information terminal determines whether the second information terminal is in the linked state, and when it is determined that the second information terminal is in the linked state, it operates in the linked operation mode, and when the first information terminal is located within a predetermined range from the second information terminal, the second information terminal determines that the second information terminal is in the linked state; It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an easy-to-use information processing technology that allows an application to be easily launched and executed on a desired device without increasing the burden on the user among multiple information terminals that have the same application or corresponding applications installed. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an information processing system according to a first embodiment. [Figure 2] 5A and 5B are explanatory diagrams for explaining a linked state in the first embodiment. [Figure 3] 5A and 5B are explanatory diagrams illustrating a non-cooperative state in the first embodiment. [Figure 4] FIG. 2 is a hardware configuration diagram of the head-mounted display according to the first embodiment. [Figure 5] FIG. 2 is a hardware configuration diagram of the smartphone according to the first embodiment. [Figure 6] 1A and 1B are functional configuration diagrams of a head-mounted display and a smartphone, respectively, according to a first embodiment. [Figure 7] 10 is a flowchart illustrating an example of application cooperation processing according to the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of application cooperation processing according to a second embodiment. [Figure 9] FIG. 11 is an explanatory diagram for explaining a cooperation status determination method according to a third embodiment. [Figure 10] 13 is a flowchart illustrating an example of application cooperation processing according to the third embodiment. [Figure 11] 10A and 10B are diagrams illustrating the display content of a display of a smartphone according to a fourth embodiment. [Figure 12] 13 is an example of a flowchart of application cooperation processing according to the fourth embodiment. [Figure 13] 13 is a flowchart illustrating an example of application cooperation processing according to the fifth embodiment. [Figure 14] 13 is a flowchart illustrating an example of application cooperation processing according to the sixth embodiment. [Figure 15] FIG. 13 is an explanatory diagram illustrating an overview of an information processing system according to a seventh embodiment. [Figure 16] 13(a) and 13(b) are explanatory diagrams for explaining a linked state in the seventh embodiment. [Figure 17] FIG. 13 is a hardware configuration diagram of a smart watch according to a seventh embodiment. [Figure 18] FIG. 13 is a functional block diagram of a smart watch according to a seventh embodiment. [Figure 19] 13 is a flowchart illustrating an example of application cooperation processing according to the seventh embodiment. [Figure 20] 10(a) and 10(b) are explanatory diagrams for explaining an overview of an information processing system according to a modified example of the present invention. [Figure 21] 10(a) to 10(e) are explanatory diagrams for explaining icon display modes according to modified examples of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] <<First Embodiment>> A first embodiment of the present invention will be described with reference to the drawings. In this embodiment, a first information processing terminal (first information terminal) and a second information processing terminal (second information terminal) execute a common application (app) in cooperation with each other.

[0010] Hereinafter, this embodiment will be described taking as an example a case where a head mounted display (hereinafter referred to as HMD) is used as the first information terminal and a smartphone (hereinafter referred to as smart phone) is used as the second information terminal.

[0011] 1 is a diagram schematically illustrating the entire information processing system 500 of this embodiment. As shown in this diagram, the information processing system 500 of this embodiment includes an HMD 100, which is a first information terminal worn on the head of a user 501, and a smartphone 200, which is a second information terminal held by the user 501.

[0012] The HMD 100 is an information terminal that embodies virtual reality (VR) technology, which creates a virtual world similar to reality and allows the user to experience the sensation of being there, and augmented reality (AR) technology, which adds digital information to the real world and reflects and extends virtual spaces (virtual objects) created using CG (Computer Graphics) and the like in the real world, and is worn on the head by a user 501. However, structural limitations limit the display range and make it difficult to perform detailed operations.

[0013] The smartphone 200 has a standalone function for making calls and communicating. The smartphone 200 displays icons on the display in an easy-to-understand manner and is suitable for touch operability. The icons are graphic symbols that can be used to launch an app by approaching or touching the icons with the user's 501 finger or a touch pen.

[0014] In this embodiment, the HMD 100 and the smartphone 200 cooperate with each other, and an instruction to start an application (hereinafter referred to as an app) installed in the HMD 100 is given from the smartphone 200, and the app is executed by the HMD 100. In this embodiment, this state is called a cooperation state. Note that hereinafter, when there is no need to distinguish between the HMD 100 and the smartphone 200, they will be called information terminals.

[0015] In the first embodiment, whether the HMD 100 and the smartphone 200 are in a linked state is determined on the HMD 100 side. Whether they are in a linked state is determined based on the arrangement state of the two. Specifically, as shown in FIG. 2(a), if an app icon 420 (421, 422, 423) displayed on the display 231 of the smartphone 200 is present within a predetermined linked field of view 101a of the display 131 (131a, 131b) of the HMD 100, it is determined that they are in a linked state.

[0016] 2(a), the icon 421 of application A (app A) in the linked visual field 101a is selected. Hereinafter, the arrangement state of the HMD 100 and the smartphone 200 in which the icon 420 of the application desired to be started is in the linked visual field 101a will be referred to as a predetermined arrangement state.

[0017] When the HMD 100 and the smartphone 200 are in a predetermined arrangement state, they are determined to be in a linked state. When the user 501 selects an icon 421 displayed on the smartphone 200 through the display 131 of the HMD 100 by touching or other action (FIG. 2(a)), for example, the smartphone 200 issues an instruction to launch an app corresponding to the selected icon 421, and the HMD 100 executes the app (FIG. 2(b)). FIG. 2(b) shows a state in which the app is executed on the HMD 100 and an icon 411 of the app is displayed.

[0018] In this embodiment, in a linked state, when the smartphone 200 accepts a selection by the user 501, such as when the user 501 touches the icon 421 of app A, communication is established between the smartphone 200 and the HMD 100, and touch information by the user 501 is transmitted from the smartphone 200 to the HMD 100.

[0019] When the HMD 100 receives touch information on the icon 421, it regards this as a start instruction and starts up the application A in the HMD 100.

[0020] As shown in FIG. 3(a), an arrangement state in which the icon 420 displayed on the display 231 of the smartphone 200 is not within the linked visual field 101a of the HMD 100 is called a non-predetermined arrangement state.

[0021] In the non-predetermined arrangement state, when the user 501 performs an action such as touching the icon 421 of the application A, and the smartphone 200 accepts the selection of the application A, the smartphone 200 starts the application A as is.

[0022] In this way, in this embodiment, even when the HMD 100 is worn, if an application is to be executed on the smartphone 200, the icon 421 can be moved outside the range of the linked visual field 101a.

[0023] In this embodiment, the HMD 100 and the smartphone 200 are each equipped with the same app or corresponding apps used to perform specific tasks such as playing music and images, creating documents, etc. Information is transmitted and received between the HMD 100 and the smartphone 200 via wireless communication or wired communication between the terminals.

[0024] [Hardware configuration] The hardware configuration of each of the HMD 100 and the smartphone 200 that constitute the information processing system 500 of this embodiment that realizes the above processing will be described below.

[0025] 4 and 5 are block diagrams respectively showing configuration examples of the HMD 100 and the smartphone 200. In these figures, the same components as those shown in FIGS. 1 to 3 are denoted by the same reference numerals.

[0026] 4 is a hardware configuration diagram of the HMD 100 of this embodiment. As shown in this diagram, the HMD 100 of this embodiment includes a main processor 111, a system bus 112, a storage device 110, an operation reception device 120, an image processing device 130, a sound processing device 140, a sensor 150, a communication device 160, an expansion interface (I / F) unit 105, a timer 106, and a vibration generator 107.

[0027] The main processor 111 is a main control unit that controls the entire HMD 100 in accordance with a predetermined program. The main processor 111 is realized by a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 111 performs overall operation control processing of the HMD 100 by executing programs such as an operating system (OS) and various operation control applications stored in the storage device 110. The main processor 111 also controls the startup operations of various applications. The main processor 111 performs processing in accordance with a clock signal measured and output by the timer 106.

[0028] The system bus 112 is a data communication path for transmitting and receiving data between the main processor 111 and each unit in the HMD 100 .

[0029] The storage device 110 includes a RAM 103 and a ROM 104. The storage device 110 stores programs such as an operating system and various operation control applications for music, images, documents, etc. The storage device 110 also stores information data such as base data required for basic operations by the operating system and file data launched by various applications. For example, when a music application installed in the HMD 100 is launched and music file data is selected, the selected music file data is played on the HMD 100, allowing the user to listen to the desired music.

[0030] The RAM 103 is a program area for executing basic operation programs and other application programs. The RAM 103 also serves as a temporary storage area for temporarily storing data as needed when various application programs are executed. The RAM 103 may be integrated with the main processor 111.

[0031] The ROM 104 stores various operational setting values of the HMD 100, information about the user of the HMD 100, and the like. The ROM 104 may also store still image data and moving image data captured by the HMD 100. The HMD 100 is also capable of expanding its functions by downloading new application programs from an application server via the Internet. At this time, the downloaded new application programs are stored in the ROM 104. The main processor 111 loads the new application programs stored in the ROM 104 into the RAM 103 and executes them, thereby enabling the HMD 100 to realize a variety of functions.

[0032] The ROM 104 is not limited to a mask ROM that can read data but cannot write data, as long as it can retain stored information even when power is not supplied to the HMD 100. The ROM 104 may be, for example, a flash ROM, a solid state drive (SSD), or a hard disk drive (HDD).

[0033] The operation reception device 120 receives input of operation instructions to the HMD 100. In this embodiment, the HMD 100 includes operation keys 121 and a touch sensor 122. The operation keys 121 include a power key, a volume key, a home key, etc. The touch sensor 122 receives operation instructions via a touchpad. Note that the HMD 100 of this embodiment does not necessarily have to include all of these operation reception devices 120. Furthermore, operations for the HMD 100 may be received via a separate information processing terminal device connected by wired or wireless communication.

[0034] The operation reception device 120 may be provided in a position and form within the HMD 100 that allows the user to easily perform input operations. The operation reception device 120 may be separated from the main body of the HMD 100 and connected by wire or wirelessly. Alternatively, the operation reception device 120 may utilize the line of sight of the user 501. For example, an input operation screen is displayed on the display 131, and input operation information is captured based on the position on the input operation screen at which the line of sight of the user 501 is directed, detected by the in-camera 134. A pointer may be displayed on the input operation screen, and the input operation information may be captured by operating the pointer. Alternatively, the user 501 may utter a voice indicating an input operation, and the voice may be collected by a microphone 143 (described later) to capture the input operation information.

[0035] The image processing device 130 is an image (video) processor, and includes a display 131, an outer camera 133, and an inner camera 134.

[0036] The display 131 is a display device (display) such as a liquid crystal panel, and presents image data processed by the image signal processing unit to the user of the HMD 100. The display 131 includes a left-eye display 131a and a right-eye display 131b. The display 131 may be an optically transmissive display or a video transmissive display.

[0037] The optically transmissive display includes a projection unit that projects various information such as playback information from the activated application and notification information for the user 501, and a transparent half mirror that forms an image of the projected information in front of the user's eyes. The video transmissive display also includes a liquid crystal panel or the like that displays various information together with real-space objects in front of the user that are photographed by the outer camera 133. Through the display 131, the user 501 can visually recognize not only the image in front of the user's eyes within the field of view, but also playback information such as music, images, and documents from the activated application.

[0038] In this embodiment, the display 131 displays the execution status of the application and the like.

[0039] The image signal processing unit is an image (video) signal processor that processes images input from outer camera 133 and inner camera 134. The image signal processing unit also superimposes an object created by main processor 111 or the like on the input image and outputs the superimposed image to display 131. The image signal processing unit may be realized by main processor 111, or may be realized by a processor dedicated to images that is provided separately from main processor 111.

[0040] The outer camera 133 acquires an image of the surroundings of the HMD 100. The outer camera 133 is installed in front of the HMD 100 and captures the forward visual field. There are two types of HMDs: an optical transmission type in which the user looks directly at a real-space object in front of the user's eyes while projecting and displaying various necessary information using a half mirror or the like, and a video transmission type in which the user combines the real-space object in front of the user photographed by the outer camera 133 with various information and displays it on a display 131 such as a liquid crystal panel. Both types capture and import the visual field in front of the user photographed by the outer camera 133.

[0041] The in-camera 134 captures an image of a different area from that of the out-camera 133. For example, the in-camera 134 captures an image of the user's eyes. In this embodiment, the in-camera 134 functions as a gaze detection sensor together with a processing unit that performs gaze identification processing.

[0042] The gaze detection sensor includes a right eye gaze detection unit and a left eye gaze detection unit, which detect the gaze of the right eye and the left eye, respectively. The gaze detection process can utilize well-known techniques commonly used for eye tracking. For example, in a method using corneal reflex, an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of the light reflected by the infrared LED (corneal reflex) is used as the reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex. In this case, an infrared camera and an infrared LED are provided.

[0043] The audio processing device 140 is an audio processor that processes audio, and includes a speaker 141 and a microphone 143.

[0044] The speaker 141 outputs the audio signal processed by the audio signal processing unit to the outside. The audio signal processing unit is an audio signal processor. The HMD 100 uses, for example, headphones.

[0045] The microphone 143 converts the user's voice and the like into audio data and inputs it. For example, an external sound microphone and a speech sound microphone may be provided. These microphones collect external sounds and the user's own speech, respectively. Headphones may also be connected. The headphones are worn on the ears of the user 501 to listen to the voices directed to the user 501.

[0046] The sensor 150 is a group of sensors for detecting the state of the HMD 100. In this embodiment, the sensor 150 includes a GPS (Global Positioning System) receiving unit 151, a gyro sensor 152, a geomagnetic sensor 153, an acceleration sensor 154, and a depth sensor 155. These sensors detect the position, movement, tilt, direction, etc. of the HMD 100. The depth sensor 155 also acquires distance information from the HMD 100 to an object. Note that other sensors may also be included.

[0047] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The gyro sensor 152 is a sensor that detects angular velocity in the rotation direction, and can detect the vertical, horizontal, and diagonal posture states. The acceleration sensor 154 and gyro sensor 152 installed in the HMD 100 can be used to detect the movement of the HMD 100.

[0048] The geomagnetic sensor 153 is a sensor that detects the magnetic force of the Earth and detects the direction in which the HMD 100 is facing. It is possible to use a three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions, and detect the movement of the HMD 100 by capturing changes in geomagnetism in response to the movement of the HMD 100. These may be used to detect and determine whether the icon 420 on the display 231 of the smartphone 200 is visible through the HMD 100.

[0049] The depth sensor 155 is a sensor that can capture the three-dimensional shape of an object such as a person or an object. It can be a LiDAR (Light Detection and Ranging) sensor that irradiates an object with infrared or other laser light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object, a TOF (Time Of Flight) sensor that measures distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel, or a millimeter-wave radar that emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The depth sensor 155 can measure the distance to, for example, a smartphone 200.

[0050] The communication device 160 is a communication interface that performs wireless communication with other information terminals, such as the nearby smartphone 200 or SW300 (described later), via short-range wireless communication, wireless LAN, or base station communication. During wireless communication, icon touch information on the smartphone 200, application information indicating an application to be launched, and file data to be played by the launched application are transmitted and received via a transmitting / receiving antenna.

[0051] The communication device 160 includes a LAN (Local Area Network) communication interface (I / F) 161, a telephone network communication I / F 162, and a BT (Bluetooth (registered trademark)) communication I / F 163. The LAN communication I / F 161 connects to an Internet wireless communication access point via wireless communication to transmit and receive data. The telephone network communication I / F 162 performs telephone communication (calls) and data transmission and reception via wireless communication with a base station of a mobile telephone communication network. The BT communication I / F 163 is an interface for communicating with an external device according to the Bluetooth standard. The LAN communication I / F 161, the telephone network communication I / F 162, and the BT communication I / F 163 each include an encoding circuit, a decoding circuit, an antenna, etc. The communication device 160 may further include an infrared communication I / F, etc.

[0052] Note that short-range wireless communication may be performed using not only Bluetooth (registered trademark), but also IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), etc. Alternatively, wireless LAN such as Wi-Fi (registered trademark) may be used. Furthermore, long-range wireless communication such as W-CDMA (Wideband Code Division Multiple Access) or GSM (Global System for Mobile communications, registered trademark) may be used for base station communication. Note that an ultra-wideband (UWB) wireless system may be used to detect the positional relationship and orientation between information terminals. This may detect and determine whether the icon 420 on the display screen of the smartphone 200 is visible through the HMD 100.

[0053] The communication device 160 may use other methods, such as optical communication or acoustic communication, as a means of wireless communication. In that case, instead of the transmitting and receiving antennas, a light emitting / receiving unit and an acoustic wave output / input unit are used, respectively.

[0054] The expansion interface unit 105 is a group of interfaces for expanding the functions of the HMD 100, and in this embodiment includes a charging terminal, a video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The video / audio interface inputs video signals / audio signals from external video / audio output devices, outputs video signals / audio signals to external video / audio input devices, etc. The USB interface connects to USB devices. The memory interface connects to memory cards and other memory media to send and receive data.

[0055] The vibration generator 107 is a vibrator that generates vibrations under the control of the main processor 111. The vibration generator 107 converts notification information for the user 501 into vibrations. The vibration generator 107 generates vibrations on the head of the user 501 who wears the HMD 100 closely, thereby ensuring that notifications are conveyed to the user 501.

[0056] 5 is a hardware configuration diagram of the smartphone 200 of this embodiment. As shown in this diagram, the smartphone 200 of this embodiment includes a main processor 201, a system bus 202, a storage device 210, an operation reception device 220, an image processing device 230, a sound processing device 240, a sensor 250, a communication device 260, an extension interface (I / F) unit 205, a timer 206, and a vibration generating device 207.

[0057] The details of each device are basically the same as those of the HMD 100. However, the smartphone 200 further includes a touch panel 223 as an operation reception device 220. The touch panel 223 includes the touch panel 123 arranged on top of the display 231. Also, instructions may be input via a keyboard or the like connected to the expansion interface unit 205.

[0058] The display 231 is configured with a liquid crystal panel or the like, and displays notification information to the user 501, such as the remaining battery capacity, various alarms, and the time, as well as the icons of apps to be launched and the execution status of the apps on the display 231. The operation reception device 220 is, for example, a touchpad-type input means such as a capacitance type, and detects approach or contact operations (touch operations) with a finger, a touch pen, or the like as operation input. The user 501 sets and inputs information that he or she wants to input, and easily selects and specifies app or file icons on the display 231 by touch operations.

[0059] [Function Block] Next, a description will be given of the functional configurations of the HMD 100 and the smartphone 200 of this embodiment. Figures 6(a) and 6(b) are functional block diagrams of the HMD 100 and the smartphone 200 of this embodiment, respectively, showing functions related to this embodiment.

[0060] As shown in FIG. 6(a), the HMD 100 includes a main control unit 170, a reception unit 171, a transmission / reception unit 172, a collaboration determination unit 173, an application execution unit 174, an output unit 175, an application program storage unit 176, and an icon data storage unit 177.

[0061] The main control unit 170 controls the operation of each unit of the HMD 100 .

[0062] The reception unit 171 receives instructions from the user 501. In this embodiment, this is realized by the main processor 111 operating the operation reception device 120. In this embodiment, for example, an instruction to start an application is received.

[0063] The transmitting / receiving unit 172 transmits and receives data to and from an external device. In this embodiment, this is realized by the main processor 111 operating the communication device 160. In this embodiment, data is transmitted and received to and from the smartphone 200 via the transmitting / receiving unit 172.

[0064] The collaboration determination unit 173 performs collaboration processing with the smartphone 200. In this embodiment, the collaboration determination unit 173 determines whether the smartphone 200 is in a collaboration state in which it can operate in a collaboration operation mode, which will be described later. In this embodiment, as shown in FIG. 2(a), the collaboration determination unit 173 displays the collaboration field of view 101a on the display 131, analyzes the image captured by the outer camera 133, and determines whether the smartphone 200 is in a predetermined arrangement state based on the displayed content.

[0065] The associated field of view 101a is set to a partial area of the field of view that the user 501 can see through the opening portion (lens portion) of the HMD 100 or the display area of the display 131. Note that the entire field of view that the user 501 can see through the HMD 100 may be set as the associated field of view 101a by changing the device configuration, setting values, etc.

[0066] As shown in FIG. 2(a), when an icon 420 of a predetermined app on the display of the smartphone 200 is shown in the captured image, the collaboration determination unit 173 determines that the smartphone 200 is in a predetermined arrangement state. The predetermined app is, for example, an app that has been notified by the smartphone 200. As will be described later, when the smartphone 200 receives that the user 501 has selected a predetermined icon 420, the smartphone 200 transmits information (app identification information) that identifies the app associated with the icon 420 to the HMD 100. Based on this app identification information, the HMD 100 identifies the icon 420 associated with the app identified by the received app identification information.

[0067] When the device is in the predetermined arrangement state, the collaboration determination unit 173 determines whether the program of the app notified from the smartphone 200 is installed on the device itself. If the program is installed, the collaboration determination unit 173 determines that the device is in the collaboration state. That is, the collaboration determination unit 173 of this embodiment determines that the device is in the collaboration state when the device is in the predetermined arrangement state and the program of the app corresponding to the icon operated by the user 501 is installed on the device itself.

[0068] If the collaboration determination unit 173 determines that the smartphone 200 is in a linked state, it transmits information indicating that the smartphone 200 is in a linked state, i.e., a linked state signal, to the smartphone 200. On the other hand, if the collaboration determination unit 173 does not determine that the smartphone 200 is in a linked state, it transmits a non-linked state signal indicating that the smartphone 200 is in a non-linked state. Note that the non-linked state signal does not have to be transmitted.

[0069] The HMD 100 stores in advance the shape data of the icon 420 of an application installed on its own terminal as icon data in the icon data storage unit 177. The icon data storage unit 177 stores the shape information of the icon 420 in association with the application identification information. The collaboration determination unit 173 refers to the icon data storage unit 177 and determines whether or not the shape data associated with the received application identification information is stored. If the shape data is stored, the collaboration determination unit 173 extracts the shape data and determines whether or not there is an image area that matches the shape data within the area corresponding to the collaboration field of view 101a.

[0070] The coordination determination unit 173 determines that the state is uncoordinated if the icon data storage unit 177 does not store shape data associated with the received application identification information. Even if shape data is stored, the coordination determination unit 173 determines that the state is uncoordinated if there is no image area matching the extracted shape data in the area corresponding to the coordinated field of view 101a of the captured image. On the other hand, the coordination determination unit 173 determines that the state is coordinated if shape data is stored and there is an image area matching the extracted shape data.

[0071] The determination of whether or not the shape corresponding to the display of the smartphone 200 and the shape of the icon 420 are present in the captured image is performed by pattern matching or the like.

[0072] Application execution unit 174 is an application execution unit that starts and executes an application when reception unit 171 receives an application start instruction. The program of the application to be executed is stored in application program storage unit 176.

[0073] The output unit 175 outputs the execution result of the executed application. In this embodiment, for example, the execution result is displayed on the display 131 or output as sound from the speaker 141.

[0074] The application program storage unit 176 stores application programs.

[0075] Shape data of icons 420 that accept execution instructions for each application program is stored as icon data in the icon data storage unit 177. The icon data is stored in association with the application. Note that the shape data is not limited to physical shapes (patterns) or images, and may also be text data or the like. When the collaboration determination unit 173 cannot determine the application from the icon pattern, it may identify the corresponding app from the accompanying text data.

[0076] The above functions are realized by main processor 111 loading a program stored in ROM 114 into RAM 113 and executing it. An application program storage unit 176 and an icon data storage unit 177 are also built in storage device 110.

[0077] Next, a description will be given of the functions of the smartphone 200 of this embodiment. FIG.

[0078] As shown in this figure, the smartphone 200 of this embodiment includes a main control unit 270, a reception unit 271, a transmission / reception unit 272, a collaboration control unit 273, an application execution unit 274, an output unit 275, an icon data storage unit 277, and an application program storage unit 276.

[0079] The main control unit 270 controls the operation of each unit of the smartphone 200.

[0080] The reception unit 271 receives instructions from the user 501. In this embodiment, this is realized by the main processor 211 operating the operation reception device 220. In this embodiment, for example, various operation instructions such as an instruction to start an application are received.

[0081] The transmission / reception unit 272 transmits and receives data to and from an external device. In this embodiment, this is realized by the main processor 211 operating the communication device 260. In this embodiment, data is transmitted and received to and from the HMD 100 via the transmission / reception unit 272.

[0082] The cooperation control unit 273 determines the operation mode of the smartphone 200 and operates the smartphone 200 in the determined operation mode. In this embodiment, the operation mode includes two types: a standalone operation mode in which the smartphone 200 executes an app by itself, and a collaborative operation mode in which the smartphone 200 executes an app in collaboration with the HMD 100.

[0083] The cooperative operation mode is an operation mode that is executed in a cooperative state. In the cooperative operation mode, the cooperation control unit 273 determines an execution device (execution terminal) that is a terminal that executes the application. However, in this embodiment, in the cooperative operation mode, the HMD 100 is determined to be the execution device.

[0084] That is, in the cooperative operation mode, when the smartphone 200 receives an instruction to start an app from the user 501 by touching the icon 420 or the like, the smartphone 200 transmits the instruction to start the app to the cooperative HMD 100. In this embodiment, the smartphone 200 does not execute the app on its own device.

[0085] In this embodiment, when the cooperation control unit 273 receives a cooperation status signal from the HMD 100, it determines that the HMD 100 is in a cooperation status and operates in a cooperation operation mode. On the other hand, when the cooperation control unit 273 receives a non-cooperation status signal, it operates in a sole operation mode.

[0086] The standalone operation mode is an operation mode in which, upon receiving an instruction from the user 501 to start an application, the application is started and executed on the device itself.

[0087] The application execution unit 274 is a second application execution unit that executes an application on the smartphone 200. In this embodiment, the application execution unit 274 is executed when the reception unit 171 receives an instruction to start the application and the cooperation control unit 273 determines that the smartphone 200 is in the standalone operation mode. The program of the application to be executed is stored in the application program storage unit 276.

[0088] The output unit 275 outputs the execution result of the application executed by the application execution unit 274. In this embodiment, the execution result is output to the display 231 and the speaker 241, for example.

[0089] Application programs are stored in the application program storage unit 276. Icons for receiving instructions to execute each application program are stored in the icon data storage unit 277. The icons are stored in association with the applications.

[0090] The above functions are realized by the main processor 211 loading a program stored in the ROM 214 into the RAM 213 and executing it. An application program storage unit 276 and an icon data storage unit 277 are also built in the storage device 210.

[0091] [Processing flow] Next, a description will be given of the flow of application cooperation processing of the information processing system 500 of this embodiment. In this embodiment, the HMD 100 and the smartphone 200 cooperate with each other to start an application of the HMD 100. Fig. 7 is a diagram illustrating the flow of application cooperation processing.

[0092] This process is started when, for example, the HMD 100 and the smartphone 200 are in a state where short-range communication is possible. In the following description of the process flow, the main control unit 170 of the HMD 100 and the main control unit 270 of the smartphone 200 will be referred to as the HMD 100 and the smartphone 200, respectively.

[0093] The smartphone 200 and the HMD 100 first establish communication with each other (S1101, S1201). In this embodiment, the communication is established by causing the respective transmission / reception units 172, 272 to transmit and receive data via the BT communication I / Fs 163, 263, for example.

[0094] The smartphone 200 displays icons 420 of apps that can be activated on the smartphone 200 on the display 231. The smartphone 200 receives a selection instruction from the user 501 (step S1202). The reception unit 271 of the smartphone 200 detects, via the touch panel 223, a touch operation on the icon 420 associated with the selected app that is displayed on the display 231. Here, it is assumed that the app A has been selected, for example.

[0095] The transmitter / receiver 272 of the smartphone 200 notifies the HMD 100 via the BT communication I / F 263 that the application A has been selected (selection information) (step S1203). The selection information includes application identification information that identifies the selected application (here, application A).

[0096] When the cooperation determination unit 173 of the HMD 100 receives the selection information of the app A from the smartphone 200 via the transmitting / receiving unit 172 (step S1102), it determines whether or not its own terminal (HMD 100) and the transmitting terminal (smartphone 200) are in a cooperation state (step S1103). Here, it determines whether or not the program of the app A is installed in its own device and whether or not the arrangement state of both terminals is a predetermined arrangement state. Either of the determinations can be made first. Then, if the program of the selected app A is installed and is in a predetermined arrangement state, it is determined that the cooperation state exists. On the other hand, in all other cases, it is determined that the cooperation state exists.

[0097] The HMD 100 returns the determination result of whether it is in a linked state or a non-linked state, that is, either a link signal or a non-linked signal, to the smartphone 200 (step S1104).

[0098] When the cooperation control unit 273 of the smartphone 200 receives the determination result via the transmission / reception unit 272 (step S1204), it determines the execution device according to the determination result (step S1205). Here, if the determination result is a cooperation state, the execution device is determined to be the HMD 100, and if the determination result is a non-cooperation state, the execution device is determined to be the smartphone 200.

[0099] The cooperation control unit 273 of the smartphone 200 notifies the HMD 100 of the determination result via the transmitting and receiving unit 272 (step S1206). In addition, the HMD 100 receives the determination result (step S1105).

[0100] Thereafter, the smartphone 200 and the HMD 100 each execute the application A in accordance with the determination result.

[0101] When the smartphone 200 is determined as the execution device (step S1207), the application execution unit 274 of the smartphone 200 starts application A in the smartphone 200 (step S1208), executes application A until application A ends (steps S1209 and S1210), and ends the process. For example, if application A is a playback application such as music, the process ends when the playback information is played to the end. Furthermore, the process may end when an end instruction is received via the reception unit 271.

[0102] On the other hand, if the execution device is not determined to be the smartphone 200, the process ends.

[0103] Similarly, when the execution device is determined to be the HMD 100 (step S1106), the application execution unit 174 of the HMD 100 starts application A in the HMD 100 (step S1107), executes application A until it is terminated (steps S1108 and S1109), and then terminates the processing. Note that an instruction to terminate application A is usually given on the HMD 100 side.

[0104] On the other hand, if the execution device is not determined to be the HMD 100, the process ends.

[0105] When application A is executed on the HMD 100, for example, if application A is an application for playing music, images, documents, etc., application A is started and executed on the HMD 100, and playback information such as music, images, documents, etc. is played back using the display 131, speaker 141, etc. If an operation to instruct the end of playback is performed via the operation reception device 120, the processing ends. Alternatively, the processing may end when the playback information has been played back to the end.

[0106] When the HMD 100 executes the application A, the instruction to end the application A may be received by the smartphone 200 and transmitted to the HMD 100. For example, the smartphone 200 continues to display the selected icon 420. After receiving the notification of the determination result in S1206, if the icon 420 is touched, the smartphone 200 determines that the instruction to end the application A has been received and transmits the instruction to end the application A to the HMD 100.

[0107] As described above, the information processing system 500 of this embodiment is an information processing system in which multiple terminals cooperate with each other to execute an application shared by the terminals. The multiple terminals include a first information terminal (HMD 100) and a second information terminal (smartphone 200). The smartphone 200 includes a reception unit 271 that receives an instruction to launch an application and a cooperation control unit 273 that causes the smartphone 200 to operate in either the independent operation mode or the cooperation operation mode. The HMD 100 also includes a cooperation determination unit 173 that determines whether the smartphone 200 is in a cooperation state in which it can operate in the cooperation operation mode, and an output unit 175 that outputs the execution result of an application executed in the cooperation operation mode. The independent operation mode is an operation mode in which an application is executed independently by the smartphone 200. The cooperation operation mode is an operation mode in which the HMD 100 and the smartphone 200 cooperate with each other to execute an application. The cooperation control unit 273 of the smartphone 200 causes the smartphone 200 to operate in the cooperation operation mode when the cooperation determination unit 173 of the HMD 100 determines that the smartphone 200 is in a cooperation state.

[0108] The linked state in which the smartphone 200 is operable in the linked operation mode is determined by the arrangement of the HMD 100 and the smartphone 200. That is, the reception unit 271 of the smartphone 200 displays an icon 420 for receiving a startup instruction on a display 231 (second display) provided in the smartphone 200, and receives the startup instruction via the icon 420. The collaboration determination unit 173 of the HMD 100 determines that the linked state is established when the icon 420 displayed on the display 231 is included within a predetermined linked field of view 101a of the display 131 of the HMD 100.

[0109] In this way, in this embodiment, when the same app or a corresponding app is installed and can be launched on the first information terminal, exemplified by HMD 100 and smartphone 200, and the second information terminal, when the app icon 420 displayed on the display 231 of the second information terminal, which is easy to operate and exemplified by smartphone 200, is touched, the information terminal that will launch the app is automatically determined depending on the arrangement of the first information terminal and the second information terminal.

[0110] Specifically, when starting an app in the HMD 100, the linked state is established by arranging the icons of the smartphone 200 through the HMD 100 in a way that is easy for the user 501 to understand. In the linked state, a desired app can be started on the HMD 100 by simply touching an icon on the smartphone 200, which is easy to operate. In other words, the app that the user wants to start on the HMD 100 can be selected on the screen of the smartphone 200, which is easy to see and operate. Therefore, according to this embodiment, the operability and usability of the HMD 100 can be significantly improved.

[0111] Furthermore, according to this embodiment, the smartphone 200, which is easy to operate, is used to start an app on the HMD 100. The smartphone 200 is placed in a predetermined position so that the app icon on the smartphone 200 can be seen through the field of view of the HMD 100, and by simply touching the app icon on the smartphone 200 in this state, the same app or a corresponding app can be started on the HMD 100 via communication.

[0112] As a result, the smartphone 200 performing a touch operation on the icon of the desired application can be seen from the HMD 100, and the user 501 can visually recognize the touch operation, so that the application can be reliably launched.

[0113] For example, if the linked visual field 101a is set in the center of the display range of the display 131, the reliability and convenience for the user 501 will be further increased.

[0114] On the other hand, according to this embodiment, if the smartphone 200's app icons are placed outside the predetermined placement state, where they are not visible through the field of view of the HMD 100, the app installed on the smartphone 200 itself can be launched even while wearing the HMD 100. For example, by viewing and touching the icon 420 in a portion of the user's field of view that is not determined to be in the predetermined placement state (a portion outside the linked field of view 101a), the app can be launched on the smartphone 200 even while wearing the HMD 100. Furthermore, even when the HMD 100 and the smartphone 200 are not linked, such as when the HMD 100 is not worn or is powered off even when worn, the app can be launched on the smartphone 200. As such, according to this embodiment, the user 501 can easily determine the device on which to launch the desired app.

[0115] According to this embodiment, simply by changing the arrangement state of the smartphone 200 and the HMD 100, it is possible to determine whether to execute the smartphone 200 and the HMD 100 in a linked operation mode in which the smartphone 200 is linked and used as an operation device for the HMD 100, or in a standalone operation mode in which the smartphone 200 is used alone, thereby reducing the burden on the user 501 and providing an information processing system 500 that is highly flexible and easy to operate.

[0116] In addition, the second information terminal may be any information terminal other than the smartphone 200 that can display icons of apps and files and allows the selection of icons, such as a tablet, smart watch, or PC.

[0117] In the above description, a music application and a video application are used as examples of applications to be executed, but the applications are not limited to these.

[0118] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, the HMD 100 determines whether the HMD 100 and the smartphone 200 are in a linked state, and the smartphone 200 determines the execution device. In this embodiment, the HMD 100 also determines the execution device.

[0119] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0120] The hardware configuration of the HMD 100 and the smartphone 200 is the same as that of the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0121] As in the first embodiment, the cooperation determination unit 173 of the HMD 100 of this embodiment determines whether the HMD 100 and the smartphone 200 are in a cooperation state or a non-cooperation state. In this embodiment, the execution device of the application A is then determined.

[0122] Specifically, if the cooperation determination unit 173 determines that the cooperation state is established, it determines the HMD 100 as the execution device, and if the cooperation determination unit 173 determines that the cooperation state is established, it determines the smartphone 200 as the execution device.

[0123] When the execution device is determined to be the HMD 100, the cooperation determination unit 173 transmits a signal (operation mode instruction signal) indicating execution in the cooperation operation mode to the smartphone 200. On the other hand, when the execution device is determined to be the smartphone 200, the cooperation determination unit 173 transmits an operation mode instruction signal to the smartphone 200 indicating execution in the independent operation mode.

[0124] Furthermore, the cooperation control unit 273 of the smartphone 200 of this embodiment operates the smartphone 200 in the determined operation mode, as in the first embodiment. However, in this embodiment, the cooperation control unit 273 does not determine the operation mode as in the first embodiment. Instead, the cooperation control unit 273 operates the smartphone 200 in the operation mode specified by the operation mode instruction signal transmitted from the HMD 100.

[0125] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below with reference to Fig. 8. The same processes as those in the application cooperation processing of the first embodiment are denoted by the same reference numerals, and repeated description will be omitted.

[0126] As in the first embodiment, the collaboration determination unit 173 of the HMD 100 of this embodiment determines whether the HMD 100 and the smartphone 200 are in a collaborative state (step S1103). After that, the collaboration determination unit 173 determines the execution device of the application A (step S2101). Here, if it is determined that the HMD 100 is in a collaborative state, it determines the execution device to be the HMD 100, and if it is determined that the HMD 100 is in a non-collaborated state, it determines the execution device to be the smartphone 200.

[0127] Then, the cooperation determination unit 173 transmits the determination result together with the determination result to the smartphone 200 (step S2102). That is, an operation mode determination signal is transmitted to the smartphone 200. Thereafter, the process of step S1106 is executed according to the determination result.

[0128] When the cooperation control unit 273 on the smartphone 200 side receives the determination result (step S2201), it executes the processes from step S1207 onward in accordance with the operation mode.

[0129] As described above, in the information processing system 500 of this embodiment, when the cooperation determination unit 173 of the HMD 100 determines that the HMD 100 is in a cooperation state, it instructs the smartphone 200 to operate in the cooperation operation mode.

[0130] As such, according to the information processing system 500 of this embodiment, as with the first embodiment, a user-friendly system can be provided that allows smooth collaboration between the HMD 100 and the smartphone 200 without increasing the burden on the user 501.

[0131] <<Third Embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the smartphone 200 determines whether or not it is in a linked state and the execution device.

[0132] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0133] The hardware configuration of the HMD 100 and the smartphone 200 is the same as that of the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0134] In this embodiment, when the collaboration determination unit 173 of the HMD 100 receives app A selection information indicating that app A has been selected from the smartphone 200, it determines whether app A is installed on the device and transmits the determination result to the smartphone 200.

[0135] In addition, when the collaboration control unit 273 of the smartphone 200 of this embodiment receives from the HMD 100 the determination result as to whether or not app A is installed, it determines whether or not the HMD 100 and the smartphone 200 itself are in a collaboration state, and determines the execution device.

[0136] The cooperation control unit 273 first determines whether the HMD 100 and the smartphone 200 are in a predetermined arrangement state. For example, as shown in FIG. 9, if the HMD 100 is present within the capturing range 201a of the in-camera 234 of the smartphone 200, it is determined that the HMD 100 is in the predetermined arrangement state. The smartphone 200 stores the shape of the HMD 100 as a pattern in advance, analyzes an image captured by the in-camera 234, and determines that the pattern is present in the image corresponding to the capturing range 201a. Note that the shape data of the HMD 100 may be transmitted from the HMD 100 to the smartphone 200 during communication and used.

[0137] When the cooperation control unit 273 determines that the HMD 100 and the smartphone 200 are in a predetermined arrangement state, it next refers to the determination result of the installation of the app A from the HMD 100. If the result received indicates that the app A is installed, it determines that the cooperation state exists. On the other hand, when the HMD 100 is not in the predetermined arrangement state, or when the HMD 100 is in the predetermined arrangement state but the app A is not installed on the HMD 100, it determines that the cooperation state exists.

[0138] Then, the cooperation control unit 273 determines the execution device and transmits the determination result of the arrangement state together with the determination result to the HMD 100. For example, if the cooperation control unit 273 determines that the state is in a cooperation state, it determines that the execution device is the HMD 100, and if the state is in a non-cooperation state, it determines that the execution device is the smartphone 200.

[0139] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below with reference to Fig. 10. The same processes as those in the application cooperation processing of the first embodiment are denoted by the same reference numerals, and repeated description will be omitted.

[0140] In this embodiment, when the collaboration determination unit 173 of the HMD 100 receives app A selection information indicating that app A has been selected from the smartphone 200 (step S1102), it determines whether app A is installed on the device itself and transmits the determination result to the smartphone 200 (step S3101).

[0141] When the cooperation control unit 273 of the smartphone 200 receives the determination result from the HMD 100 (step S3201), the cooperation control unit 273 determines the cooperation state (step S3202). Here, as described above, the cooperation control unit 273 determines whether or not both devices are in a predetermined arrangement state based on the determination result received from the HMD 100 (step S3202).

[0142] Then, the cooperation control unit 273 determines the execution device according to the determination result of the cooperation state (step S3203), and transmits the determination result to the HMD 100 together with the determination result of the cooperation state.

[0143] Thereafter, the processes from step S1207 onwards are executed depending on the determination result.

[0144] Furthermore, on the HMD 100 side, when the judgment result and the decision result are received from the smartphone 200 (step S3102), the HMD 100 executes the processes from step S1106 onwards according to the decision result.

[0145] As described above, in the information processing system 500 of this embodiment, the cooperation control unit 273 of the smartphone 200 determines whether the smartphone 200 is in a cooperation state in which the smartphone 200 can operate in the cooperation operation mode. If it is determined that the smartphone 200 is in a cooperation state, the cooperation control unit 273 causes the smartphone 200 to operate in the cooperation operation mode.

[0146] As such, according to the information processing system of this embodiment, as with the first embodiment, a user-friendly information processing system can be provided that allows smooth collaboration between the HMD 100 and the smartphone 200 without increasing the burden on the user 501.

[0147] <<Fourth Embodiment>> Next, a fourth embodiment of the present invention will be described. In this embodiment, even if the execution device of the selected application A is determined to be the HMD 100, the application A is also executed on the smartphone 200 side, and an operation for the application A from the user 501 is made available.

[0148] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0149] The hardware configuration of the HMD 100 and the smartphone 200 is the same as that of the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0150] When the collaboration control unit 273 on the smartphone 200 side determines that the smartphone 200 is in a collaboration state, even if the execution device is determined to be the HMD 100, the smartphone 200 side also executes the app A in synchronization with the HMD 100.

[0151] For example, as shown in Fig. 11(a), when the icon 421 of app A is selected, the cooperation control unit 273 transmits a start-up instruction to the HMD 100 and also causes the smartphone 200 to execute app A. Then, as shown in Fig. 11(b), an operation screen showing that app A is being executed is also displayed on the display 231 of the smartphone 200. Here, a case where app A is a music playback app is shown as an example.

[0152] In this embodiment, when the execution device is determined to be HMD100 and the collaboration control unit 273 receives an operation for app A, such as pause, skip, back, or stop, via the display 231 of the smartphone 200, the collaboration control unit 273 transmits a signal (operation signal) indicating the received operation to the HMD100.

[0153] When the cooperation determination unit 173 of the HMD 100 receives an operation signal from the smartphone 200 via the transmission / reception unit 172 while the application A is being executed, the cooperation determination unit 173 controls the operation of the application A in accordance with the operation signal.

[0154] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below with reference to Fig. 12. The same processes as those in the application cooperation processing of the first embodiment are denoted by the same reference numerals, and repeated description will be omitted.

[0155] In this embodiment, the method of any of the first to third embodiments may be used for the process of determining whether the HMD 100 and the smartphone 200 are in a linked state and determining the execution device accordingly. Here, this process is referred to as an execution device determination process.

[0156] In step S4400, the cooperation control unit 273 of the smartphone 200 performs an execution device determination process, and when the execution device is determined, instructs the application execution unit 274 to launch application A. In response to this, the application execution unit 274 launches application A (step S4201).

[0157] Then, processing is performed depending on whether the execution device is a smartphone (step S1207). That is, if the execution device is determined to be the smartphone 200, processing from step S1209 onwards is performed as in the first to third embodiments.

[0158] On the other hand, if the execution device is determined to be the HMD 100, the application execution unit 274 on the smartphone 200 side also executes application A in synchronization with the HMD 100 (step S4202). Then, when the reception unit 271 receives an operation for application A via the display 231 or the like (step S4203), the transmission / reception unit 272 transmits an operation signal corresponding to the operation to the HMD 100. This continues until application A is terminated. FIG. 12 illustrates, as an example, a case where an operation to terminate application A is received.

[0159] Furthermore, on the HMD 100 side, even if application A is being executed, when an operation signal is received from the smartphone 200 via the transmission / reception unit 172, the application execution unit 174 processes the operation signal as an operation signal for application A. For example, as shown in FIG. 12, when an operation signal instructing termination is received (step S4101), application A is terminated (step S4102) and the processing is terminated.

[0160] As described above, in the information processing system 500 of this embodiment, even in the cooperative operation mode, the smartphone 200 also executes the app synchronously. Therefore, while the app is running on the HMD 100, the smartphone 200 can accept an operation on the app at any time.

[0161] That is, according to the present embodiment, when the smartphone 200 and the HMD 100 are in a predetermined arrangement state, touching the icon of app A on the smartphone 200 causes app A to be executed on the HMD 100. At this time, the smartphone 200 transitions to an operation screen for app A, and accepts operations for app A being executed on the HMD 100. For example, touching the pause button on the operation screen displayed on the smartphone 200 pauses playback of app A being executed on the HMD 100.

[0162] Thus, according to the information processing system of this embodiment, similarly to the first embodiment, it is possible to provide a user-friendly information processing system that can smoothly link the HMD 100 and the smartphone 200 without increasing the burden on the user 501. Furthermore, according to this embodiment, even when an app is running on the HMD 100, operations for the app can be performed on the smartphone 200, which further improves usability and convenience.

[0163] It is not necessary for the smartphone 200 to operate the entire app. For example, the smartphone 200 may be configured to operate only the I / F portion, transmit an operation signal corresponding to the received operation to the HMD 100, and perform subsequent processing on the HMD 100 side. For example, the smartphone 200 may display operation buttons, receive operation input via the operation buttons, and transmit the operation signal to the HMD 100. The app on the HMD 100 then performs playback in accordance with the operation signal. This allows the smartphone 200's resources to be used efficiently, and allows the HMD 100's app to be operated on an easy-to-use screen on the smartphone 200.

[0164] <<Fifth Embodiment>> Next, a fifth embodiment of the present invention will be described. In this embodiment, when the HMD 100 is determined to be the execution device in the linked state, the smartphone 200 transmits data required for executing application A to the HMD 100 before starting application A on the HMD 100. This embodiment is an embodiment in which application A is, for example, an application for playing a file.

[0165] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0166] The hardware configuration of the HMD 100 and the smartphone 200 is the same as that of the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0167] In this embodiment, when the execution device is determined to be the HMD 100 in the cooperative state, that is, when in cooperative operation mode, the cooperation control unit 273 of the smartphone 200 transmits data necessary for executing the selected application A to the HMD 100.

[0168] After receiving the data transmitted from the smartphone 200, the cooperation determination unit 173 of the HMD 100 starts up and executes the application A on the HMD 100.

[0169] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below with reference to Fig. 13. The same processes as those in the application cooperation processing of the first embodiment are denoted by the same reference numerals, and repeated description will be omitted.

[0170] In this embodiment, the method of any of the first to third embodiments may be used for the process of determining whether the HMD 100 and the smartphone 200 are in a linked state and determining the execution device accordingly. Here, this process is referred to as an execution device determination process.

[0171] In step S4400, the cooperation control unit 273 of the smartphone 200 performs the execution device determination process, and when the execution device is determined, controls the operation of the smartphone 200 accordingly.

[0172] That is, if the execution device is determined to be the HMD 100, the cooperation control unit 273 causes the transmission / reception unit 272 to transfer data necessary for processing the application A to the HMD 100 (step S5201), and ends the processing. On the other hand, if the execution device is determined to be the smartphone 200, the processing from step S1208 onwards is executed.

[0173] Also, on the HMD100 side, when the execution device is determined to be HMD100, when data is received from smartphone 200 (step S5101), the collaboration determination unit 173 causes the application execution unit 174 to launch application A (step S1107) and executes the processing from step S1108 onwards.

[0174] As described above, in the information processing system 500 of this embodiment, when the smartphone 200 is operated in a collaborative operation mode, the collaboration control unit 273 transmits data required to execute the app to the HMD 100, and the HMD 100 executes the app using the data received from the smartphone 200.

[0175] As such, according to the information processing system of this embodiment, as with the first embodiment, a user-friendly information processing system can be provided that allows smooth collaboration between the HMD 100 and the smartphone 200 without increasing the burden on the user 501.

[0176] Furthermore, in this embodiment, data necessary for executing an app is stored in the smartphone 200, and does not need to be constantly stored in the HMD 100, for which it is desirable to reduce storage capacity such as memory. Therefore, the HMD 100 can execute an app while keeping storage capacity low.

[0177] Furthermore, for example, if the application is a playback application for music, image documents, etc., the data stored in the smartphone 200 can be played on the HMD 100 using the playback application installed in the HMD 100. In other words, this data can be played on a more familiar device, which is more convenient.

[0178] In the present embodiment, the case where the application to be executed is first identified and then the file data is transferred has been described as an example, but this is not limiting. The application to be executed may not be limited, and only the file data may be transferred. In this case, the HMD 100 checks the received file data and starts an application that processes the file data.

[0179] Furthermore, when the same file is stored in the smartphone 200 and the HMD 100, the file data may not be transferred, but only the file specification information may be transmitted, and the file in the HMD 100 may be played on the HMD 100. This can prevent delays in playback due to data transfer and reduce power consumption.

[0180] <<Sixth Embodiment>> Next, a sixth embodiment of the present invention will be described. In this embodiment, even in a linked state, an application is executed on the smartphone 200 side, and only the execution result is transmitted to the HMD 100.

[0181] The present embodiment will be described below, focusing on the configuration that differs from the first embodiment.

[0182] The hardware configuration of the HMD 100 and the smartphone 200 is the same as that of the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0183] In the cooperative state, even when the execution device is determined to be the HMD 100, the cooperation control unit 273 of the smartphone 200 executes the application A on the smartphone 200 side. However, when the HMD 100 is the execution device, the cooperation control unit 273 transmits the execution result to the HMD 100 instead of outputting it to the smartphone 200 itself.

[0184] When the output device is determined to be the HMD 100, the cooperation determination unit 173 of the HMD 100 controls the output unit 175 to output the execution result (output information) of the application A received via the transmission / reception unit 172. Note that if a screen, processing, or the like specific to the application A is required to output the execution result, the application A is also started up in the HMD 100 to perform the processing.

[0185] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below with reference to Fig. 14. The same processes as those in the application cooperation processing of the first embodiment are denoted by the same reference numerals, and repeated description will be omitted.

[0186] In addition, in this embodiment, it is determined whether the HMD 100 and the smartphone 200 are in a linked state, and accordingly, an output device that is the output destination of the application A is determined. This output device determination process is the same as the execution device determination process that determines the execution device in any one of the first to third embodiments, but determines an output device instead of an execution device.

[0187] In step S6400, the cooperation control unit 273 of the smartphone 200 performs output device determination processing, and when the output device is determined, instructs the application execution unit 274 to start application A. In response to this, the application execution unit 274 starts application A (step S1208). Then, the cooperation control unit 273 causes the application execution unit 274 to execute application A (step S1209).

[0188] When outputting, it is determined whether the output device determined in the output device determination process is the smartphone 200 or the HMD 100 (step S6201). If the output device is the smartphone 200, the cooperation control unit 273 causes the output unit 275 to output directly to the display 231, speaker 241, etc. of its own device (step S6202). The processes of steps S1209 to S6202 are continued until application A is terminated, and then the process is terminated.

[0189] On the other hand, if it is determined that the output device is the HMD 100, when outputting, the cooperation control unit 273 causes the transmitting / receiving unit 272 to transmit output information, which is information to be output, to the HMD 100 (step S6203).

[0190] On the HMD 100 side, if the output device is determined to be the smartphone 200, the processing is terminated (step S6101), and if the output device is determined to be the HMD 100, the collaboration determination unit 173 prepares to receive output information by instructing the transmission / reception unit 172 to receive output information (step S6102).

[0191] Then, when output information is received from the smartphone 200 (step S6103), the collaboration determination unit 173 causes the output unit 175 to output the received output information (step S6104). The collaboration determination unit 173 executes the processes of S6103 and S6104 until reception of the output information is completed (step S6105), and then terminates the process. Note that whether or not the output has completed may be determined by, for example, receiving an output information completion signal, a specified time having elapsed since the output information could no longer be received, or communication with the smartphone being interrupted.

[0192] As described above, in the information processing system 500 of this embodiment, when the smartphone 200 is operated in the collaborative operation mode, the collaboration control unit 273 of the smartphone 200 executes the selected app and transmits the execution result to the HMD 100.

[0193] As such, according to the information processing system of this embodiment, as with the first embodiment, a user-friendly information processing system can be provided that allows smooth collaboration between the HMD 100 and the smartphone 200 without increasing the burden on the user 501.

[0194] In particular, according to this embodiment, when the user 501 touches the icon 420 of a music app in a predetermined position, for example, while looking at the smartphone 200 through the HMD 100, music software is executed on the smartphone 200, and only audio playback information is transmitted to the HMD 100, allowing the user to listen to the music through the speakers (headphones) of the HMD 100. This eliminates the need for the HMD 100 to include extra information data memory or apps for large-capacity content data such as video and music. In other words, it is possible to reduce the amount of storage required, while still allowing the HMD 100 to operate as if an app such as music had been launched and played.

[0195] This reduces the storage capacity of the HMD 100 and also reduces the required processing power, thereby enabling the HMD 100 to be made smaller and lighter. Furthermore, according to this embodiment, even when an operation is being executed on the HMD 100, the operation can be performed on the smartphone 200, which improves usability.

[0196] <<Seventh Embodiment>> Next, a seventh embodiment of the present invention will be described. As shown in Fig. 15, an information processing system 500 of this embodiment includes a first information terminal, a second information terminal, and a third information terminal, and these three information terminals cooperate with each other to execute an application.

[0197] Hereinafter, this embodiment will be described using an example in which the first information terminal is a smart watch (SW300), the second information terminal is a smartphone 200, and the third information terminal is an HMD 100.

[0198] In this embodiment, the HMD 100 determines whether the smartphone 200 and the SW 300 are in a position where they can cooperate with each other. Then, based on the result of the determination, the smartphone 200 determines whether to execute the selected application A on its own device or on the SW 300, i.e., whether to operate the selected application A in a cooperative operation mode or a standalone operation mode. When the smartphone 200 determines the operation mode, it checks whether the selected application A is installed in the SW 300 in advance.

[0199] The configuration of each device in this embodiment will be described below. The hardware configuration of the HMD 100 and the smartphone 200 is the same as that in the first embodiment. The functional configuration is also the same. However, the processing of the cooperation determination unit 173 and the cooperation control unit 273 is different.

[0200] The hardware configuration and functional blocks of the HMD 100 and the smartphone 200 of this embodiment are basically the same as those of the first embodiment, except for the processing of the cooperation determination unit 173 and the cooperation control unit 273.

[0201] The HMD 100 of this embodiment only determines whether or not the third information terminal is in a position where cooperation is possible. That is, the cooperation determination unit 173 of this embodiment determines whether or not the smartphone 200 and the SW 300 are in a position where cooperation is possible.

[0202] 16(a), when the screen of the smartphone 200 displaying the icon 420 and the SW300 are both present within the linked field of view 101a of the display 131, it is determined that the smartphone 200 and the SW300 are in a linked state. If the program of the app corresponding to the icon 420 is installed in the SW300, the selected app is executed in the SW300, as shown in FIG. 16(b).

[0203] The hardware configuration of SW300 of this embodiment is shown in Fig. 17. As shown in this figure, SW300 of this embodiment includes a main processor 311, a system bus 312, a storage device 310, an operation reception device 320, an image processing device 330, a sound processing device 340, a sensor 350, a communication device 360, an extension interface (I / F) unit 305, a timer 306, and a vibration generating device 307.

[0204] The details of each device are basically the same as those of the same name in the HMD 100. However, the SW 300 further includes a touch panel 323 as an operation reception device 320. The touch panel 323 is disposed on top of a display 331.

[0205] [Function Block] Next, functional blocks of SW300 are shown in Fig. 18. As shown in this figure, SW300 includes a main control unit 370, a reception unit 371, a transmission / reception unit 372, a cooperation control unit 373, an application execution unit 374, an output unit 375, an icon data storage unit 377, and an application program storage unit 376.

[0206] Each functional block has the same function as the configuration of the same name in the HMD 100. Note that the cooperation control unit 373 of the SW 300, like the cooperation control unit 273 of the smartphone 200, determines whether or not the SW 300 operates in a cooperation operation mode in which the SW 300 cooperates with another information terminal to execute an app. If it determines that the SW 300 is in the cooperation operation mode, it causes the app execution unit 374 to start the app in response to a start instruction from the other information terminal.

[0207] [Processing flow] The flow of application cooperation processing in the information processing system 500 of this embodiment will be described below. Fig. 19 shows the processing flow of application cooperation processing of this embodiment. The same processes as in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0208] First, the HMD 100 and the smartphone 200, and the SW 300 and the smartphone 200 establish communication with each other (steps S1101, S1201, S1301). As in the first embodiment, the respective transmission / reception units 172, 272, and 372 perform data communication to establish communication.

[0209] The smartphone 200 receives an instruction to select the application A (step S1202).

[0210] The smartphone 200 notifies the HMD 100 and the SW 300 of the selection information of the application A (step S1203).

[0211] The cooperation determination unit 173 of the HMD 100 determines whether the smartphone 200 and SW300 are in a position where cooperation is possible using the above method (step S7101), and transmits the determination result to the smartphone 200 (step S7102). Note that communication may also be established between the HMD 100 and SW300, and the determination result may also be transmitted to SW300.

[0212] In addition, upon receiving notification of the application A selection information from the smartphone 200, the collaboration control unit 373 of the SW300 determines whether application A is installed on the device itself and transmits the result to the smartphone 200 as application A information (step S7303).

[0213] The cooperation control unit 273 of the smartphone 200 receives the determination result of the placement state from the HMD 100 and the application A information from the SW 300 (step S7201). Using this information, the cooperation control unit 273 determines whether the smartphone 200 and the SW 300 are in a cooperation state, and determines the execution device for executing the application A (step S7202). In this embodiment, if the information obtained indicates that the placement state is cooperation-enabled and that the application A is installed in the SW 300, the execution device is determined to be the SW 300. In other cases, the execution device is determined to be the smartphone 200.

[0214] The cooperation control unit 273 of the smartphone 200 notifies the HMD 100 and the SW 300 of the determination result (step S7203). Upon receiving the determination result, the HMD 100 ends the process.

[0215] After that, the smartphone 200 executes the processes from step S1207 onwards.

[0216] Furthermore, if SW300 determines that the execution device is SW300 (step S7305), it starts up application A in SW300 (step S7306), executes application A until application A terminates (steps S7307, S7308), and terminates processing.

[0217] On the other hand, if the execution device is not determined to be SW300, the process ends.

[0218] As described above, the information processing system 500 of this embodiment includes the HMD 100, the smartphone 200, and the SW 300 as multiple terminals. The HMD 100 determines whether the smartphone 200 is in a linked state, in which it can operate in the linked operation mode. If the smartphone 200 is determined to be in a linked state, it operates in the linked operation mode and transmits an instruction to start the selected app to the SW 300. The SW 300 receives the start instruction and starts and executes the app.

[0219] As described above, according to this embodiment, an application on the SW300 can be launched by the smartphone 200, which is easy to operate. Furthermore, cooperation between the two is realized by determining the positional relationship between the two through the field of view of the HMD 100, which is a third information terminal. When an application is executed on the SW300, the operability can be improved. In other words, this embodiment realizes a highly convenient information terminal cooperation technology without burdening the user 501.

[0220] Furthermore, according to this embodiment, when starting an app, if an easy-to-operate smartphone 200 is used and positioned so that the icon of the smartphone 200 and the SW300 are visible through the field of view of the HMD 100, the desired app can be started on the SW300 by touching the icon on the smartphone 200 in this state. Therefore, the display unit and operation unit of the SW300 can be simple. In other words, the configuration of the SW300 can be simplified.

[0221] In this embodiment, when the smartphone 200 is in a predetermined positional state in which it is within the linked field of view 101a of the HMD 100 and the SW 300 is not in the linked field of view 101a of the HMD 100, the execution device may be determined to be the HMD 100, as in the first embodiment. In this case, the smartphone 200 transmits a start instruction to the HMD 100, and the application A is started within the HMD 100.

[0222] Furthermore, in this embodiment, the first information terminal is not limited to SW 300. It may be, for example, a tablet, a PC, or another smartphone 200 different from the second information terminal, as long as it is an information terminal that can launch an app and easily optimize the usage position.

[0223] In this embodiment, the SW300 may execute an app on the smartphone 200 in synchronization with the SW300, and operations may be accepted by the smartphone 200, as in the fourth embodiment. Also, as in the fifth embodiment, the SW300 may receive data transfer from the smartphone 200 before executing the app. Furthermore, as in the sixth embodiment, the app may be executed on the smartphone 200 side, and only the execution results may be received and output.

[0224] By transferring file data stored on smartphone 200 to SW300 and playing the data using a file playback app installed on SW300, SW300 does not need to store all file data at all times, which makes it possible to reduce data volume, which is important for a small, multi-functional wristwatch-type information terminal like SW300.

[0225] <Variation 1> In each of the above embodiments, whether or not the two information terminals are in a predetermined positional state is determined based on whether or not the two information terminals are in a predetermined positional relationship. That is, it is determined based on whether or not the two information terminals are in a positional state such that the icon 420 on the display 231 of the smartphone 200 is present within the linked visual field 101a of the HMD 100. However, the determination method is not limited to this.

[0226] For example, an ultra-wideband (UWB) system may be used to detect the positional relationship and orientation between information terminals, and the results may be used to identify whether the smartphone 200 and HMD 100 are in a predetermined arrangement state. Alternatively, the movement of each information terminal may be determined from output information of various sensors 150, 250 mounted on the smartphone 200 and HMD 100, respectively, and the positional relationship and orientation between the terminals may be detected and used to identify whether the smartphone 200 and HMD 100 are in a predetermined arrangement state.

[0227] Furthermore, whether or not the predetermined arrangement state is established may be determined based on, for example, the distance between two information terminals. For example, it may be determined based on the distance between the HMD 100 and the smartphone 200. That is, if the distance between the two is less than a predetermined distance threshold, it is determined to be established in the predetermined arrangement state.

[0228] The distance between them is measured by the depth sensor 155 of the HMD 100 in the first, second, and seventh embodiments, for example. Also, in the third embodiment, it is measured by the depth sensor 255 of the smartphone 200.

[0229] In particular, when determining whether or not the smartphone 200 is in a predetermined arrangement state based on the distance between the two devices, the process is simple. That is, a desired app can be launched on the HMD 100 by touching an icon using the smartphone 200 located in the vicinity within a certain range, without worrying about whether the smartphone 200 is within the associated field of view 101a of the HMD 100. Also, by intentionally moving the smartphone 200 away by more than a certain distance, the app can be launched on the smartphone 200 side. This further improves convenience. Also, compared to a process that uses the associated field of view 101a for determination, there is a reduction in erroneous operations, such as when the smartphone 200 was intended to be removed from the field of view but was not.

[0230] Alternatively, the linked visual field 101a may not be determined, and the line of sight direction of the user 501 may be detected, and if the icon 420 of the smartphone 200 overlaps a predetermined area on the display 131 that includes the intersection with the line of sight direction, the predetermined arrangement state may be established.

[0231] In this case, even if the icon 420 of the smartphone 200 is visible on the display 131 while wearing the HMD 100, the app will not start on the HMD 100 unless the user 501 is looking at it. Therefore, a device with higher operability can be provided.

[0232] Furthermore, the associated visual field 101a may be set, and when the icon 420 is within the range of the associated visual field 101a and the user 501 is aligning his / her line of sight with the icon 420, it may be determined that the icon 420 is in the predetermined arrangement state.

[0233] <Variation 2> In the above embodiment, when it is determined that the devices are in a linked state, the application is started on a predetermined information terminal, but the method for determining the device on which the application is to be executed is not limited to this.

[0234] For example, after it is determined that the devices are in a linked state, the device on which the app is to be executed may be determined according to the manner of the selection operation on the icon 420 displayed on the smartphone 200. For example, if the icon 420 is single-tapped, the app is started on the smartphone 200, and if the icon 420 is double-tapped, the app is started on the HMD 100.

[0235] Alternatively, whether or not the device is in a linked state may be determined using a different method, and only the execution device may be determined based on the line of sight of the user 501. The line of sight of the user 501 may be detected, and if the line of sight is not on the icon 420 or the display 231 of the smartphone 200, the HMD 100 may not start the app even if the device is in a linked state.

[0236] This allows the user to intentionally launch an app on the smartphone 200 instead of the HMD 100 by keeping the smartphone 200 within their field of vision while looking away from the smartphone 200 and operating the icon.

[0237] Furthermore, the execution device may be determined not only by the gaze direction but also by whether the user 501 has opened or closed their eyes. In this case, images acquired by the in-camera 134 over a predetermined period of time are analyzed to identify whether the eyes are closed for a certain specified time, which is different from blinking and allows for determining whether the eyes are closed intentionally. For example, if the icon 420 is touched with the eyes closed for a certain specified time, it is determined that the app should be launched on the smartphone 200.

[0238] Also, if the user intentionally closes his / her eyes when tapping the icon and releasing his / her finger, the smartphone 200 may be activated. This is effective when the user wants to intentionally and explicitly activate the smartphone 200 while using the HMD 100 all the time.

[0239] Alternatively, the smartphone 200 may be configured to start up when one eye is closed. In this case, it becomes possible to easily perform a desired operation while looking at the icon 420 with the other eye.

[0240] <Variation 3> In each of the above embodiments, data transmission and reception between the information terminals is not limited to direct wireless communication between them. For example, as shown in Figures 20(a) and 20(b), data may be transmitted via a server 610.

[0241] Similarly, the application, files, and data may be stored on the server 610 and transferred via communication to each information terminal as appropriate in response to an instruction from the smartphone 200. In response to an instruction from the smartphone 200, the application may be started on the server 610, and playback information may be transferred to an information terminal determined in each embodiment for viewing.

[0242] The server 610 may be of any form as long as it can connect to each information terminal via communication and send and receive necessary information, and may be, for example, a local server, a cloud server, an edge server, or a network service.

[0243] In addition to transferring large-volume content data from the smartphone 200 to the HMD 100, content data may also be downloaded to the HMD 100 from a cloud network. In particular, image apps that handle high-definition video and the like use a significantly large amount of data. This significantly reduces the capacity of the HMD 100. In this case, using a high-speed, large-capacity communication network such as 5G (5th Generation: 5th Generation mobile communication system) local 5G for data communication can further improve usability.

[0244] <Variation 4> Furthermore, a terminal mark (hereinafter referred to as a small icon) may be added to a predetermined position on the display screen of the icon 420 touched by the user 501 as a small graphic that indicates on which information terminal the application or file can be launched. Examples of icons that include small icons are shown in Figures 21(a) to 21(c). Here, as an example, a music playback application is shown as application A, and its icon 430 is shown.

[0245] 21(a) is an example of an icon 430 having a small icon 431 indicating that the app can be launched by the HMD 100 within the icon 430. Also, FIG. 21(b) is an example of an icon 430 having a small icon 432 indicating that the app A can be moved by the SW 300.

[0246] 21(c) shows an example of an application that can be activated only by the smartphone 200. Here, an icon 440 of a spreadsheet application is shown as an example. No small icon is displayed in a predetermined position within the icon 440 of the spreadsheet application.

[0247] As a result, when touching an icon on the smartphone 200, the user can visually check the status of the icon they are about to touch, including whether or not there are any small icons such as those shown in Figures 21(a), 21(b), and 21(c), and know in advance which information terminal will be able to launch the icon when touched, thereby easily improving usability.

[0248] The small icons described in Figures 21(a), 21(b), and 21(c) indicate information terminals that can be activated when the information terminals are in a specified arrangement state, regardless of whether they are in a specified arrangement state or not.

[0249] Furthermore, when the cooperation control unit 273 of this embodiment receives information from the HMD 100 or SW 300 that the respective information terminals have application A, it displays the small icon 431 at a predetermined position of the icon 430. Alternatively, information as to whether or not an application program is installed may be exchanged in advance between the HMD 100, SW 300, and smartphone 200, and the small icon 431 may be displayed.

[0250] 21(d) and 21(e) show other examples of the small icon 431. The small icon 431 may be displayed in a manner that allows identification of the information terminal that launches an app or file among the linked information terminals when the icon 420 is touched. In other words, the display manner of the small icon 431 may be changed depending on whether or not the information terminals are linked.

[0251] For example, if the smartphone 200 and HMD 100 are in a linked state and touching the icon 420 launches an app or file on the HMD 100, a highlighted (shaded) small icon 432 is displayed as shown in Fig. 21(d). On the other hand, if the smartphone 200 and HMD 100 are not in a linked state and an app is launched on the smartphone 200, a small icon 433 without highlighting (shading) is displayed as shown in Fig. 21(e).

[0252] In this way, by displaying the small icon 431, the user 501 can easily grasp the information terminals that can be activated, and can thereby clearly select the information terminal to be activated at the user's will.

[0253] As an identification and notification method other than displaying the small icon 431, the information terminal on which the application can be activated may be notified to the user 501 through the speaker 141 (headphones) as audio information.

[0254] Furthermore, the HMD 100 may generate vibration information that identifies which information terminals can launch an app based on the magnitude or type of vibration, and notify the user 501 using the vibration generator 107. This provides even greater convenience for the user 501.

[0255] Whether or not to display small icons may be specified as a default setting for each app when the app is installed. Alternatively, the user 501 may be able to change the default setting. Alternatively, the display determination for all apps may be made collectively (by the OS or the like) on a device-by-device basis, rather than on an app-by-app basis, and the user 501 may be able to switch the setting at his / her command.

[0256] <Variation 5> In the above-described embodiments and modified examples, the icon touch detection method is a method in which the operation reception device 120 of the smartphone 200 detects the icon touch operation. However, the present invention is not limited to this. For example, the icon touch operation on the smartphone 200 may be detected by capturing an image of the icon touch operation on the smartphone 200 with the outer camera 133 of the HMD 100 and analyzing the captured image. In addition, as a method for identifying the icon position, the HMD 100 may transmit coordinates on the display 231 of the smartphone 200 captured by the outer camera 133 of the HMD 100 to the smartphone 200, and the smartphone 200 may identify the app located at those coordinates and notify the HMD 100 of the coordinates.

[0257] <Variation 6> Furthermore, in the above embodiments, the HMD 100 and the smartphone 200 are given as examples of information terminals, but any information terminal capable of acquiring arrangement information between multiple information terminals may be used. For example, both may be smartphones. In this case, the smartphone 200A displays the icon 420 and accepts touches, and the icon may be touched while the smartphone 200B captures an image with its camera, thereby starting an app on the smartphone 200B.

[0258] The same application may be operated in cooperation between the information terminals. For example, when the same map application is used on the smartphone 200 and the HMD 100, if both terminals are in a predetermined position, a detailed map may be displayed on the smartphone 200 and a simplified map may be displayed on the HMD 100, and so on. By operating the applications in cooperation, usability can be improved.

[0259] <Variation 7> Furthermore, when the same app or a corresponding app is installed on one information terminal, it may be installed synchronously, such as by installing it on the other information terminal. If the same app or a corresponding app as the app selected on one information terminal is not available on the other information terminal, it may be downloaded from a server or from another information terminal, installed, and made available for use. Furthermore, linking between information terminals (pairing, group setting, etc.) may be performed in advance. When communication is established with an information terminal that is not linked, the user 501 may be asked whether or not to link, and if the user 501 indicates his or her intention to link, the linking may be performed at that time.

[0260] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0261] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0262] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0263] 100: HMD, 101a: linked field of view, 103: RAM, 104: ROM, 105: extended interface unit, 106: timer, 107: vibration generator, 110: storage device, 111: main processor, 112: system bus, 113: RAM, 114: ROM, 120: operation reception device, 121: operation key, 122: touch sensor, 123: touch panel, 130: image processing device, 131: display, 131a: left eye display, 131b: right eye display, 133: outer camera, 134: inner camera camera, 140: audio processing device, 141: speaker, 143: microphone, 150: sensor, 151: receiving unit, 152: gyro sensor, 153: geomagnetic sensor, 154: acceleration sensor, 155: depth sensor, 160: communication device, 161: LAN communication I / F, 162: telephone network communication I / F, 163: BT communication I / F, 170: main control unit, 171: reception unit, 172: transmission / reception unit, 173: cooperation determination unit, 174: application execution unit, 175: output unit, 176: application program storage unit, 177: icon data storage unit, 200: Smartphone, 200A: Smartphone, 200B: Smartphone, 201: Main processor, 201a: Shooting range, 202: System bus, 205: Expansion interface unit, 206: Timer, 207: Vibration generator, 210: Storage device, 211: Main processor, 213: RAM, 214: ROM, 220: Operation reception device, 223: Touch panel, 230: Image processing device, 231: Display, 234: In-camera, 240: Audio processing device, 241: Speaker, 250: Sensor, 255: Depth sensor, 260: Communication device, 263: Bluetooth communication I / F, 270: Main control unit, 271: Reception unit, 272: Transmitting / receiving unit, 273: Collaboration control unit, 274: Application execution unit, 275: Output unit, 276: Application program storage unit, 277: Icon data storage unit, 300: Smart watch, 305: Expansion interface unit, 306: Timer, 307: Vibration generator, 310: Storage device, 311: Main processor, 312: System bus, 320: Operation reception device, 323: Touch panel, 330: Image processing device, 331: Display, 340: Audio processing device, 350: Sensor, 360: Communication device, 370: Main control unit, 371: Reception unit, 372: Transmitting / receiving unit, 373: Collaboration control unit, 374: Application execution unit, 375: Output unit, 376: Application program storage unit, 377: Icon data storage unit, 411: Icon, 420: Icon, 421: Icon, 430: Icon, 431: Small Icon, 432: Small Icon, 433: Small Icon, 440: Icon, 500: Information Processing System, 501: User, 610: Server

Claims

1. Information processing that links multiple devices and executes applications that are shared by each device 1. A system comprising: The plurality of terminals include a first information terminal and a second information terminal, The second information terminal receiving an instruction to start the application; operating the second information terminal in either a standalone operation mode or a linked operation mode; The independent operation mode is an operation in which the application is executed independently on the second information terminal. This is the production mode. The cooperative operation mode is a mode in which the application is executed by the first information terminal and the second information terminal. This is an operation mode that is executed in cooperation with the terminal, The first information terminal outputting an execution result of the application executed in the collaborative operation mode; The linked state in which the second information terminal is operable in the linked operation mode is determined by the arrangement of the information terminal and the second information terminal, The second information terminal The second information terminal determines whether it is in the linked state, and if it determines that it is in the linked state, It works in handheld mode, When the first information terminal is within a predetermined range from the second information terminal, the second information terminal is in the linked state. To judge that An information processing system characterized by:

2. 2. The information processing system according to claim 1, The first information terminal determining whether the device is in the linked state; The second information terminal When the first information terminal determines that it is in the linked state, it operates in the linked operation mode. 、 An information processing system characterized by:

3. 2. The information processing system according to claim 1, The first information terminal determining whether the device is in the linked state; When it is determined that the second information terminal is in the linked state, the second information terminal is instructed to operate in the linked operation mode. Giving instructions, An information processing system characterized by:

4. 2. The information processing system according to claim 1, The plurality of terminals further includes a third information terminal, The third information terminal is in a position where the second information terminal can operate in the linked operation mode. Determine whether there is The second information terminal When the second information terminal is determined to be in the operable position, it operates in the linked operation mode. To create, An information processing system characterized by:

5. 2. The information processing system according to claim 1, When the second information terminal is operated in the cooperative operation mode, Sending a start-up instruction to the first information terminal; The first information terminal receives an instruction to start the application from the second information terminal. and then executing the application. An information processing system characterized by:

6. 6. The information processing system according to claim 5, the second information terminal displays an execution state of the application being executed by the first information terminal; executing the application in synchronization with the first information terminal; An information processing system characterized by:

7. 2. The information processing system according to claim 1, When the second information terminal is operated in the cooperative operation mode, Transmitting data required for execution to the first information terminal; The first information terminal receives from the second information terminal the Upon receiving the data, executing the application using the data; An information processing system characterized by:

8. 2. The information processing system according to claim 1, When the second information terminal is operated in the cooperative operation mode, the application and transmitting the execution result to the first information terminal. An information processing system characterized by:

9. 5. An information processing system according to claim 2, 3, or 4, the second information terminal displays an icon for accepting the start-up instruction; Accepting the start instruction via the icon; The first information terminal detects the icon displayed by the second information terminal within a predetermined cooperation range. If the link is included, it is determined that the link is in the linked state. An information processing system characterized by:

10. 2. The information processing system according to claim 1, Whether or not the second information terminal is in the linked state is determined based on the distance between the second information terminal and the first information terminal. To refuse, An information processing system characterized by:

11. 2. The information processing system according to claim 1, the second information terminal displays an icon for accepting the start-up instruction; Accepting the start instruction via the icon; The icon is a terminal that can start an application associated with the icon. The device has a terminal mark that identifies the type of device. An information processing system characterized by:

12. 12. The information processing system according to claim 11, The terminal mark indicates that the application is being executed on the terminal identified by the terminal mark. If so, change the display mode. An information processing system characterized by:

Images