Information processing terminal

Abstract

To easily and conveniently establish application coordination among a plurality of information processing terminals.SOLUTION: An information processing terminal is provided with: a communication unit for transmitting and receiving information to / from other terminals; a detection unit for detecting a predetermined cooperation mark; a cooperation unit for, when transmitting a notification indicating that the cooperation mark has been detected to the other terminal upon detecting the cooperation mark and receiving the notification indicating the detection of the cooperation mark from the other terminal, performing management to execute an application in cooperation with the other terminal as a constituent terminal of a cooperation group; and an execution unit for executing the application associated with the cooperation mark in cooperation with the other terminal. The detection unit detects a mark of a predetermined application as the cooperation mark when the information processing terminal determines that it is in a predetermined arrangement state.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for supporting the cooperative execution of applications (hereinafter referred to as apps) among a plurality of information processing terminals. [Background technology]

[0002] It is becoming increasingly common for multiple information terminals to execute applications in cooperation with each other. For example, Patent Document 1 discloses an information processing system as a technology for selecting an optimal cooperation destination device to cooperate with a cooperation source device, which includes: "a cooperation source device, multiple cooperation destination devices connectable to the cooperation source device, and an application cooperation apparatus that controls cooperation between apps registered in the cooperation source device and the multiple cooperation destination devices, wherein the application cooperation apparatus includes: a first management unit that manages device information including information used to connect to the cooperation destination devices and information regarding data used by apps registered in the cooperation destination devices; a second management unit that, in response to a cooperation request from the cooperation source device, refers to the device information and acquires cooperation candidates indicating cooperation destination devices and apps that can be cooperated with; and a cooperation control unit that launches the apps indicated by the cooperation candidates and determines, as cooperation destinations, devices to perform cooperation and apps registered in the devices that can be cooperated with (abstract excerpt)."

[0003] There is also a technology for cooperatively controlling multiple devices based on a determination of the identity of images taken by the multiple devices. For example, Patent Document 2 discloses a determination device "including a first acquisition unit that acquires a first image taken by a first device together with the shooting time, a second acquisition unit that acquires a second image taken by a second device together with the shooting time, a determination unit that determines the identity of the shooting positions of the first image and the second image based on the captured image and shooting time of each of the first image and the second image, and an output unit that outputs the determination result to the first device or the second device (abstract excerpt)." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-142693 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-174253 Summary of the Invention [Problem to be solved by the invention]

[0005] 2. Description of the Related Art Users wearing information terminals such as head mounted displays (HMDs) are increasingly forming groups to share and run applications installed on the information terminals.

[0006] Although Patent Document 1 describes establishing cooperative communication and launching an application on all information terminals that can launch the application, it does not suggest anything about selecting an information terminal to launch the application, which makes it difficult to form a group among users and share an application.

[0007] Furthermore, although Patent Document 2 describes linking devices based on the identity of captured images, it does not disclose anything about launching an app or linking apps.

[0008] The present invention has been made in view of the above circumstances, and has an object to provide a technique for realizing application linkage between a plurality of information processing terminals in a simple and user-friendly manner. [Means for solving the problem]

[0009] The present invention is an information processing terminal for executing an application, the information processing terminal including a communication unit for transmitting and receiving information to and from other terminals, a detection unit for detecting a predetermined linkage mark, and a communication unit for detecting the linkage mark by the detection unit. ,before The link mark was detected. and a first terminal identification information for identifying the information processing terminal. notification data of The communication unit Send to the other terminal ,before Other terminals but The link mark was detected. and second terminal identification information that identifies the other terminal. 2nd notice data of The communication unit receives the message from the other terminal. When receiving the message, the other terminal in cooperation with the information processing terminal application Realize A collaboration unit manages the application associated with the collaboration mark as a constituent terminal of the collaboration group that performs the collaboration. The data obtained by the processing of the application is launched. The other terminal managed by the collaboration unit as a constituent terminal of the collaboration group and receiving data relating to the processing of the application associated with the link mark from the other terminal, In collaboration with The application associated with the link mark and an execution unit that executes the link mark, and when the detection unit determines that the information processing terminal is in a predetermined arrangement state, the detection unit detects a mark of a predetermined application as the link mark. [Effects of the Invention]

[0010] According to the present invention, it is possible to realize application linkage between a plurality of information processing terminals in a simple and user-friendly manner. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an information processing system according to a first embodiment. [Figure 2] FIG. 1 is a configuration diagram of an information processing system according to a first embodiment. [Figure 3] FIG. 2 is a hardware configuration diagram of the HMD according to the first embodiment. [Figure 4] 1A is a functional block diagram of an HMD according to a first embodiment, FIG. 1B is an explanatory diagram illustrating an example of an application list according to the first embodiment, FIG. 1C is an explanatory diagram illustrating an example of notification data according to the first embodiment, and FIG. 1D is an explanatory diagram illustrating an example of a collaboration group list according to the first embodiment. [Figure 5] 5(a) to 5(c) are explanatory diagrams for explaining detection of a linkage mark according to the first embodiment. [Figure 6] 10 is a flowchart of a linking process according to the first embodiment. [Figure 7] FIG. 10 is a configuration diagram of an information processing system according to a second embodiment. [Figure 8] (a) is a functional block diagram of a server of the second embodiment, (b) is an explanatory diagram for explaining an example of a collaboration group list of the second embodiment, and (c) is a hardware configuration diagram of a server of the second embodiment. [Figure 9] 10 is a flowchart of a linking process according to a second embodiment. [Figure 10] FIG. 10 is a system configuration diagram of a modified example of the second embodiment. [Figure 11] FIG. 10 is an explanatory diagram for explaining a system configuration of a third embodiment. [Figure 12] 10A and 10B are explanatory diagrams illustrating detection of a linkage mark by a detection unit according to a third embodiment. [Figure 13] FIG. 10 is a hardware configuration diagram of a smartphone according to a third embodiment. [Figure 14] FIG. 10A is a functional block diagram of a smartphone according to a third embodiment, and FIG. 10B is an explanatory diagram for explaining an example of icon display according to a modified example of the third embodiment. [Figure 15] 10 is a flowchart of a linking process according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] <<First Embodiment>> An example of a first embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is an explanatory diagram for explaining an overview of an information processing system according to this embodiment.

[0013] In recent years, virtual reality (VR) technology, which creates a virtual world similar to reality and allows users to experience the sensation of being there, and augmented reality (AR) technology, which adds digital information to the real world and reflects and augments the real world with virtual spaces (virtual objects) created using computer graphics (CG), have been used in games and various simulations. Head-mounted displays (HMDs), which are devices equipped with a display and camera, are widely used as information processing devices that embody these technologies. Users wearing information processing devices such as HMDs form groups and collaborate to run apps installed on the devices, enabling them to chat, hold meetings, play games, and more.

[0014] It is desirable that these applications be executed in a linked manner easily and safely, and this embodiment achieves this.

[0015] In this embodiment, multiple users 900, each wearing a head-mounted display (HMD) 100, cooperate to execute a common application (app) installed in each HMD 100. The apps installed in each HMD 100 are the same or equivalent and are used to perform specific tasks such as chatting, playing music and images, and creating documents.

[0016] The collaboration is realized when the user 900 recognizes a mark corresponding to the application to be executed. The mark is, for example, a graphic for calling up a function such as launching an application, and may be printed or written somewhere, or may be displayed on the screen of an information processing terminal or the like. The mark is not limited to a geometrical graphic, and may be a character or the like. Any mark may be used as long as it can determine the application to be launched, and will be referred to as a collaboration mark 810 hereinafter. Furthermore, the application launched on each HMD 100 upon recognition of the collaboration mark 810 does not need to be the same application. For example, the application launched when a chat mark is recognized may be an application that allows users to chat in collaboration with each other.

[0017] 1 illustrates HMDs 101, 102, 103, and 104 respectively worn on the heads of four users 901, 902, 903, and 904, and a linkage mark 810. When there is no need to distinguish between them, they are represented by an HMD 100 and a user 900.

[0018] Information is transmitted and received between the HMDs 100 via wireless communications 912, 913, 914, 923, 924, and 934. Note that information may be transmitted and received between the HMDs 100 via a server or the like, rather than via direct wireless communications between the HMDs 100.

[0019] In this figure, HMDs 101, 102, and 103 detect that users 901, 902, and 903 have recognized a linkage mark 810 in their respective lines of sight 941, 942, and 943, and identify the app corresponding to the linkage mark 810 as an app to be executed in linkage (linkage app).

[0020] Furthermore, the HMDs 101, 102, and 103 are HMDs 100 that have detected the same coordination mark 810, and are identified as a coordination group that coordinates and executes a coordinated application via wireless communications 912, 913, and 923. Then, the HMDs 101, 102, and 103 identified as the coordination group coordinate and execute a coordinated application.

[0021] 1, the linked app corresponding to the link mark 810 is a chat app. In this case, the HMDs 101, 102, and 103 execute the corresponding chat app in a linked manner. A chat screen showing the content of the conversation between each user 900 is displayed on the display 131 of the HMDs 101, 102, and 103.

[0022] On the other hand, the user 904 of the HMD 104 whose line of sight 944 is not directed toward the coordination mark 810 does not recognize the coordination mark 810. That is, the HMD 104 does not detect the coordination mark 810. Therefore, the HMD 104 does not identify the app corresponding to the coordination mark 810 as a coordinated app. Therefore, the HMD 104 does not join a coordination group that coordinates and executes coordinated apps. Therefore, a chat screen showing the content of the conversation between the users 900 is not displayed on the display 131 of the HMD 104.

[0023] For example, in a chat app, it is usually necessary to select a chat partner or a chat group after launching the app. However, in this embodiment, when a user 900 who wants to participate simply looks at a collaboration mark 810 associated with the chat app, the chat app is automatically launched, and the HMDs 100 of users 900 who are viewing the same collaboration mark 810 are recognized as a collaboration group. As described above, according to this embodiment, a collaboration group in which only the HMDs 100 worn by users 900 who are viewing the same collaboration mark 810 are used to execute a collaboration app in collaboration is automatically formed. Furthermore, the collaboration app is identified simply by the user 900 wearing the HMD 100 recognizing (detecting) the collaboration mark 810. This eliminates the need to select members of the collaboration group or launch a collaboration app.

[0024] Furthermore, the HMD 100 of a user 900 who does not see the collaboration mark 810 does not join the members of the collaboration group. Therefore, the HMD 100 of an unintended user 900 is not added to the members of the collaboration group by mistake. That is, only the HMD 100 of a user 900 who is present at the location joins the collaboration group. Therefore, there is no risk of accidentally sending information to the HMD 100 of a user 900 who is not present at the location, and the collaboration group can be formed safely. Furthermore, even in a situation where the users 900 are frequently replaced, the constituent members of the collaboration group can be easily managed.

[0025] [System Configuration] The system configuration of the information processing system 700 of this embodiment that realizes the above will be described below. Fig. 2 is a system configuration diagram of the information processing system 700 of this embodiment.

[0026] As shown in the figure, the information processing system 700 of this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a coordination mark 810.

[0027] The HMD 100 is a device worn on the head of a user 900, and displays processed information on a display 131. The HMD 100 of this embodiment has a shape of glasses, and includes the display 131 at the lens position of the glasses. The display 131 may be a transmissive type or a non-transmissive type.

[0028] In the HMD 100, the user 900 can observe the situation in real space via the display 131. In addition, an augmented reality (AR) object can be displayed on the display 131. Therefore, the user 900 can simultaneously view both the AR object displayed on the display 131 and the situation in real space.

[0029] As described above, the linkage mark 810 is a mark that, when detected, serves as a reference for identifying the linked app and certifying the members of the linked group. As described above, for example, an icon figure of the linked app is used. In this embodiment, the linkage mark 810 is fixed to a predetermined position, such as a wall, a pillar, or a desk.

[0030] [HMD] Next, the hardware configuration and functional blocks of the HMD 100 of this embodiment will be described with reference to the drawings.

[0031] [Hardware configuration] The hardware configuration of the HMD 100 of this embodiment that realizes the above processing will be described below. Figure 3 is a block diagram showing an example of the configuration of the HMD 100 of this embodiment. In these figures, the same components as those shown in Figures 1 and 2 are assigned the same reference numerals.

[0032] As shown in the figure, the HMD 100 of this embodiment includes a main processor 111, a system bus 112, a storage device 110, an operation reception device 120, an image processing device 130, a sound processing device 140, a sensor 150, a communication device 160, an expansion interface (I / F) unit 171, a timer 172, and a vibration generator 173. Each component is connected to each other via the system bus 112, except for the transmitting and receiving antenna.

[0033] The main processor 111 is a main control unit that controls the entire HMD 100 in accordance with a predetermined program. The main processor 111 is realized by a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 111 performs overall operation control processing of the HMD 100 by executing programs such as an operating system (OS) and various operation control applications stored in the storage device 110. The main processor 111 also controls the startup operations of various applications. The main processor 111 performs processing in accordance with a clock signal measured and output by a timer 172.

[0034] The system bus 112 is a data communication path for transmitting and receiving data between the main processor 111 and each unit in the HMD 100 .

[0035] The storage device 110 includes a RAM 113, a ROM 114, and a flash memory 115. The storage device 110 stores programs such as an operating system and various operation control applications for music, images, documents, etc. It also stores information data such as base data required for basic operations by the operating system and file data launched by various applications. For example, when a music application installed in the HMD 100 is launched and music file data is selected, the selected music file data is played on the HMD 100, allowing the user to listen to the desired music.

[0036] The RAM 113 is a program area for executing basic operation programs and other application programs. The RAM 113 also serves as a temporary storage area for temporarily storing data as needed when various application programs are executed. The RAM 113 may be integrated with the main processor 111.

[0037] The ROM 114 and flash memory 115 store various programs for realizing the functions of the HMD 100, operation setting values, sensor information including detection values ​​from a sensor 150 (described later), and various display data such as virtual objects and content.

[0038] Flash memory 115 stores operation programs downloaded from network 300 and various data created by the operation programs. Each operation program stored in flash memory 115 can be updated and its functions expanded by downloading it from each server device on network 300.

[0039] Furthermore, flash memory 115 can store content such as moving images, still images, and audio downloaded from network 300. It also stores data such as moving images and still images captured by in-camera 134 or out-camera 133.

[0040] The ROM 114 and flash memory 115 are so-called non-volatile storages that retain stored information even when power is not supplied from an external source to the HMD 100. Hereinafter, unless there is a need to distinguish between the ROM 114 and the flash memory 115, they will be collectively referred to as the internal memory storage. The main processor 111 loads new application programs stored in the internal memory storage into the RAM 113 and executes them, thereby enabling the HMD 100 to realize a variety of functions.

[0041] The built-in memory storage needs to retain stored information even when power is not supplied to the HMD 100. Therefore, devices such as flash ROM, SSD (Solid State Drive), and HDD (Hard Disc Drive) are used as the built-in memory storage.

[0042] The built-in memory storage stores programs such as the operating system and various operation control applications for chat, music, images, documents, etc. It also stores information data such as base data required for basic operations by the operating system and file data launched by various applications.

[0043] For example, when a chat application installed in the HMD 100 is started and various file data necessary for chat operations, such as chat partners and previous chat content, is selected, the selected chat file data can be used to converse with the desired chat partner on the HMD 100. Also, when a music application installed in the HMD 100 is started and music file data is selected, the selected music file data is played on the HMD 100, allowing the user to listen to the desired music.

[0044] The operation reception device 120 receives input of operation instructions for the HMD 100. In this embodiment, the HMD 100 includes operation keys 121. The operation keys 121 include a power key, a volume key, a home key, and the like.

[0045] The HMD 100 of this embodiment does not necessarily have to include all of these operation reception devices 120. The operation reception device 120 may be provided in a position and form within the HMD 100 that allows the user 901 to easily perform input operations.

[0046] The operation reception device 120 may also receive operations of the HMD 100 via a separate information processing terminal device connected via wired or wireless communication. In this case, the device may include input means such as a keyboard, key buttons, or touch keys. The operation reception device 120 may also utilize the gaze of the user 901. For example, an input operation screen is displayed on the display 131, and input operation information is captured based on the position on the input operation screen at which the user 901's gaze is directed, detected by an in-camera 134 (described later). A pointer may be displayed on the input operation screen, and the input operation information may be captured by operating the pointer. Alternatively, the user 901 may utter a voice indicating an input operation, which may be collected by a microphone 143 (described later), and the input operation information may be captured. The out-camera 133 and various sensors 150 may also be used to detect hand or body movements (gestures) of the user 901 and capture the input operation information.

[0047] The image processing device 130 is an image (video) processor, and includes a display 131, an outer camera 133, and an inner camera 134.

[0048] The display 131 is a display device (display) such as a liquid crystal panel, and presents image data processed by the image signal processing unit to the user of the HMD 100. The display 131 includes a display for the left eye and a display for the right eye. The display 131 may be a transmissive (optically transmissive) display or a non-transmissive (video transmissive) display.

[0049] The optically transmissive display includes a projection unit that projects various information such as playback information from the activated application and notification information for the user 901, and a transparent half mirror that forms an image of the projected information in front of the user's eyes. The video transmissive display also includes a liquid crystal panel or the like that displays various information together with real-space objects in front of the user that are photographed by the outer camera 133. Through the display 131, the user 901 can visually recognize not only the image in front of the user's eyes within the field of view, but also playback information such as chat conversations, music, images, and documents from the activated application.

[0050] The display 131 displays notification information to the user, such as the remaining battery capacity, various alarms, and the time, and also displays an icon of an application to be started within the display 131.

[0051] The image signal processing unit is an image (video) signal processor that processes images input from outer camera 133 and inner camera 134. The image signal processing unit also superimposes an object created by main processor 111 or the like on the input image and outputs the superimposed image to display 131. The image signal processing unit may be realized by main processor 111, or may be realized by a processor dedicated to images that is provided separately from main processor 111.

[0052] The outer camera 133 and the inner camera 134 are cameras that input image data of the surroundings and objects by converting light input from the lens into an electrical signal using electronic devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0053] The outer camera 133 captures an image of the surroundings of the HMD 100. The outer camera 133 is installed in front of the HMD 100 and captures the forward visual field. The HMD 100 is classified into an optical transmission type and a video transmission type depending on the type of display 131, but both types capture and capture the visual field in front of the user's eyes, captured by the outer camera 133.

[0054] The in-camera 134 captures an image of a different area from that of the out-camera 133. For example, it captures an image of the eyes of the user 901. In this embodiment, the in-camera 134 functions as a gaze detection sensor together with a processing unit that performs gaze identification processing.

[0055] The gaze detection sensor includes, for example, a right eye gaze detection unit and a left eye gaze detection unit, which detect the gaze of the right eye and the left eye, respectively. The gaze detection process can utilize well-known techniques commonly used for eye tracking processing. For example, in a method using corneal reflex, an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of ​​the reflected light from the infrared LED (corneal reflex) is used as a reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex. In this case, an infrared camera and an infrared LED are provided.

[0056] The operation reception device 120 may display an input operation screen within the display screen of the display 131, and capture input operation information based on the position on the input operation screen at which the gaze detected by the right eye gaze detection unit and the left eye gaze detection unit is directed. A pointer may be displayed on the input operation screen, and the operation reception device 120 may operate the pointer to capture input operation information.

[0057] The audio processing device 140 is an audio processor that processes audio, and includes a speaker 141 and a microphone 143.

[0058] The speaker 141 outputs the audio signal processed by the audio signal processing unit to the outside. The speaker 141 outputs audio to the outside to notify the user of audio such as notification information and music. The audio signal processing unit is an audio signal processor. The HMD 100 uses, for example, headphones.

[0059] The microphone 143 collects the user's voice and the like, converts it into audio data, and inputs it. For example, an ambient sound microphone and a speech sound microphone may be provided. These microphones collect external sounds and the user 901's own speech, respectively. Headphones may be connected. The headphones are worn on the ears of the user 901 and listen to the voices directed to the user 901. The headphones can notify the user 900 of notification information by voice. The operation reception device 120 may also be configured such that the user 900 speaks a voice indicating an input operation, and the voice is collected by the microphone 143 to capture the input operation information.

[0060] The sensor 150 is a group of sensors for detecting the state of the HMD 100. In this embodiment, the sensor 150 includes a GPS (Global Positioning System) receiver 151, a gyro sensor 152, a geomagnetic sensor 153, an acceleration sensor 154, and a depth sensor 155. These sensors detect the position, movement, tilt, direction, etc. of the HMD 100. The depth sensor 155 also acquires distance information from the HMD 100 to an object. Note that other sensors may also be included.

[0061] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The gyro sensor 152 is a sensor that detects angular velocity in the rotation direction, and can detect the vertical, horizontal, and diagonal posture states. The acceleration sensor 154 and gyro sensor 152 installed in the HMD 100 can be used to detect the movement of the HMD 100.

[0062] The geomagnetic sensor 153 is a sensor that detects the magnetic force of the earth and detects the direction in which the HMD 100 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions may be used, and the movement of the HMD 100 may be detected by capturing changes in the geomagnetism in response to the movement of the HMD 100.

[0063] The depth sensor 155 is a sensor that can capture the three-dimensional shape of an object such as a person or an object. It can be a LiDAR (Light Detection and Ranging) sensor that irradiates an object with infrared or other laser light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object, a TOF (Time Of Flight) sensor that measures distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel, or a millimeter wave radar that emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object.

[0064] The communication device 160 is a communication interface that performs wireless communication with other nearby information terminals via short-range wireless communication, wireless LAN, or base station communication. During wireless communication, data is transmitted and received via a transmitting / receiving antenna. In this embodiment, the communication device 160 transmits and receives participation ID information, which is a collection of all IDs of each information terminal participating in the cooperative execution of an application, application launch instruction information, file data to be played by the launched application, and the like.

[0065] The communication device 160 includes a LAN (Local Area Network) communication interface (I / F) 161, a telephone network communication I / F 162, and a short-range wireless communication I / F 163.

[0066] The LAN communication I / F 161 is connected to a network 300 such as the Internet via a wireless access point or the like, and transmits and receives data to and from each server on the network 300. The connection to the wireless access point or the like may be made using Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0067] The telephone network communication I / F 162 is connected to a communication network using a mobile telephone communication network, and transmits and receives information to and from a server on the communication network. The communication methods used include, for example, third-generation mobile communication systems (hereinafter referred to as "3G") such as the GSM (registered trademark) (Global System for Mobile Communications) method, the W-CDMA (Wideband Code Division Multiple Access) method, the CDMA2000 method, and the UMTS (Universal Mobile Telecommunications System) method, as well as communication methods called the LTE (Long Term Evolution) method, the fourth generation (4G), and the fifth generation (5G).

[0068] The short-distance wireless communication I / F 163 is an interface for short-distance wireless communication. Short-distance wireless communication is a communication method for directly transmitting and receiving data between surrounding HMDs 100. In this embodiment, this is performed using, for example, an electronic tag. The electronic tag is a communication method that uses RFID (Radio Frequency Identification) technology to read and write data in an IC chip without contact using radio waves.

[0069] However, the short-distance wireless communication is not limited to this, and may be any method that allows wireless communication at least when the HMD 100 is near another information terminal. For example, communication according to the Bluetooth standard, infrared communication using IrDA (Infrared Data Association, registered trademark) or the like, or communication using Zigbee (registered trademark) or HomeRF (Home Radio Frequency, registered trademark) may be used. Alternatively, communication may be performed using a wireless LAN such as Wi-Fi (registered trademark). Furthermore, an ultra-wideband wireless system (UWB) may be used.

[0070] The LAN communication I / F 161, the telephone network communication I / F 162, and the short-range wireless communication I / F 163 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0071] The communication device 160 may use other methods, such as optical communication or acoustic communication, as a means of wireless communication. In that case, instead of the transmitting and receiving antennas, a light emitting / receiving unit and an acoustic wave output / input unit are used, respectively.

[0072] Furthermore, in the case of image applications that handle high-definition video, the amount of data used is dramatically large, and in such cases, high-speed, large-capacity communication networks such as 5G (5th Generation: 5th generation mobile communication system) and local 5G may be used for wireless communication. This can prevent display delays due to increased data volume and quality degradation due to data loss during transmission, improving usability.

[0073] The expansion interface unit 171 is a group of interfaces for expanding the functions of the HMD 100, and in this embodiment includes a charging terminal, a video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The video / audio interface inputs video signals / audio signals from external video / audio output devices, outputs video signals / audio signals to external video / audio input devices, etc. The USB interface connects to USB devices. The memory interface connects to memory cards and other memory media to send and receive data.

[0074] The vibration generator 173 generates vibrations under the control of the main processor 111, and converts notification information sent to the user by the HMD into vibrations. The vibration generator 173 generates vibrations on the user's head, which is worn closely, so that notifications can be reliably conveyed to the user.

[0075] 3 includes components that are not essential to this embodiment, but the effects of this embodiment are not impaired even if these components are not provided. Furthermore, components not shown, such as a digital broadcast reception function and an electronic money payment function, may be further added.

[0076] [HMD functional blocks] Next, the functional configuration of the HMD 100 of this embodiment will be described. Fig. 4(a) is a functional block diagram of the configuration of the HMD 100 according to this embodiment. As shown in this diagram, the HMD 100 includes a main control unit 210, a detection unit 220, an execution unit 230, and a collaboration unit 240. In addition, the HMD 100 includes an application list 250, a collaboration group list 260, and application / processing data 270 as data used for processing.

[0077] The main control unit 210 controls the overall operation of the HMD 100. By executing programs such as an operating system (OS) and various operation control applications, the main control unit 210 controls the overall operation of the HMD and also controls the launch and execution of various applications.

[0078] The detection unit 220 detects the linkage mark 810 associated with the application. In this embodiment, as shown in Fig. 5(a) and Fig. 5(b), the detection unit 220 captures an image of the periphery (field of view) of the HMD 100 with the outer camera 133 and analyzes the resulting image (captured image 136). If the captured image contains an area 137 that matches, to a predetermined extent, a figure registered in association with the application in the application list 250, the detection unit 220 determines that the linkage mark 810 has been detected.

[0079] Here, the application list 250 will be described. The application list 250 is a list in which data (mark data) of the linkage mark 810 is stored in association with an application. In this embodiment, as shown in FIG. 4(b), mark data 252 is registered in association with application identification information (application ID (identification) 251) that is information for identifying an application. The mark data 252 includes, for example, mark shape data 253 and mark identification information (mark ID 254) that is information for identifying the mark.

[0080] The application list 250 is registered, for example, when an application is installed. The mark data 252 may be, for example, icon data used to start an application. The mark data 252 may be updated (changed or added) when an application is updated. The mark data 252 is not limited to one type per application.

[0081] When the HMD 100 is not executing an association application, the detection unit 220 analyzes the captured image 136 at predetermined time intervals and determines whether the captured image 136 includes an area 137 corresponding to the shape data 253 registered in the application list 250. The determination of whether the captured image 136 includes an area 137 (figure) corresponding to the association mark 810 is performed by pattern matching or the like. If the area 137 is included, it is determined that the association mark 810 has been detected. Note that the detection operation by the detection unit 220 may be started by an instruction from the user 900, for purposes such as reducing power consumption or not wanting to perform association even if the association mark 810 is recognized.

[0082] When the linkage mark 810 is detected, the detection unit 220 identifies the application registered in the application list 250 in association with the detected mark data 272 as an application to be executed in a linked manner (linkage application).

[0083] The detection unit 220 may perform the above analysis when it receives a mark detection instruction from the user 900.

[0084] Furthermore, for example, when a mark that is not stored as mark data 272 in the application list 250 is extracted from the captured image 136, the detection unit 220 may inquire of a server on the network and determine whether or not there is an application associated with the mark. If there is an application associated with the mark, the application is downloaded together with the mark, and the mark is stored in the application list 250 in association with the application ID 251. The application itself is stored in the application processing data 270.

[0085] In addition, when detecting, in order to determine whether the linkage mark 810 corresponds to a linkage app or is simply a graphic, criteria may be set in advance for the linkage mark 810, such as size, shape, type of frame, etc.

[0086] 5(c), whether or not the detection has been successful may be determined based on whether or not the associated mark 810 is present in a predetermined area (associated field of view 131a) within the field of view of the outer camera 133. The associated field of view 131a may be determined in advance, or may be a predetermined range centered on the intersection of the line of sight of the user 900 and the field of view of the outer camera 133.

[0087] The execution unit 230 starts and executes the application identified as the cooperative application by the detection unit 220. The application is stored in the application processing data 270 in association with the application ID 251 identified by the detection unit 220. When the application is executed, it is executed in cooperation with other HMDs 100.

[0088] The application / processing data 270 is the application and various file data required to execute the application. Using this data, the execution unit 230 can launch and execute a desired linked application. During the linked execution, the execution unit 230 transmits and receives information between multiple HMDs 100 via the communication device 160.

[0089] The collaboration unit 240 manages the HMDs 100 that have detected the same collaboration mark 810 as members of a collaboration group. In this embodiment, the collaboration unit 240 manages the relevant HMDs 100 among the HMDs 100 that are within the communication range of short-range wireless communication and can communicate with each other.

[0090] When a change occurs in the state (status) of the HMD 100, the collaboration unit 240 generates notification data notifying the change and transmits it to surrounding HMDs 100. In this embodiment, for example, when the collaboration unit 240 detects the collaboration mark 810, notification data indicating the detection is generated. Furthermore, when the collaboration unit 240 terminates a running collaboration app, notification data indicating the termination is generated.

[0091] Here, the notification data 280 generated by the linking unit 240 will be described. As shown in FIG. 4(c), the notification data 280 registers a mark ID 281, which is information for identifying the detected linking mark 810, terminal identification information (device ID 282), which is information for identifying the HMD 100 itself, an address 283 of the HMD 100, an application ID 284, which is information for identifying the linked application, and information indicating the status (here, detected or terminated; status 285). The device ID 262 of each device of the HMD 100 is set in advance. The address 283 is used, for example, when linking applications is performed via a network such as a LAN. The address 283 may be used as the device ID 282.

[0092] For example, when the detection unit 220 detects the linkage mark 810, the notification data 280 (hereinafter, detection notification data 280a) in which "detected" is registered in the status 285 is generated and transmitted. On the other hand, when the linkage app is terminated, the notification data 280 (hereinafter, termination notification data 280b) in which "terminated" is registered in the status 285 is generated and transmitted. The notification data 280 is transmitted, for example, by short-range wireless communication.

[0093] In this embodiment, the detection notification data 280a also serves as inquiry data inquiring whether the same coordination mark 810 has been detected. When the coordination unit 240 receives detection notification data 280a from another HMD 100 in which the mark ID 281 is the same as the mark ID 254 of the coordination mark 810 detected by the own device, the coordination unit 240 returns the detection notification data 280a to the HMD 100 that sent the data to indicate that the own device has also detected the same coordination mark 810. In this way, when joining a coordination group midway, information on all HMDs 100 that are already participating in the group can be obtained.

[0094] Furthermore, when the collaboration unit 240 detects the collaboration mark 810, it generates and manages a collaboration group list 260. As shown in FIG. 4(d), the collaboration group list 260 is a list that manages the HMDs 100 that execute the collaboration app in collaboration as members of a collaboration group. The collaboration group list 260 includes a mark ID storage location 261 that stores the mark ID of the collaboration mark, and device IDs 262 and addresses 263 of the HMDs 100 in the collaboration group.

[0095] When the collaboration mark 810 is detected, the collaboration unit 240 first generates the collaboration group list 260 and registers the mark ID (A) corresponding to the detected collaboration mark 810 in the mark ID storage location 261 of the collaboration group list 260. After that, when the collaboration unit 240 receives detection notification data 280a for the same mark ID (A) as the mark ID (A) registered in the mark ID storage location 261 of the collaboration group list 260, the collaboration unit 240 registers the device ID 282 and address 283 included in the detection notification data 280a as the device ID 262 and address 263 of the collaboration group list. Furthermore, when the collaboration unit 240 receives completion notification data 280b for the same mark ID (A) as the mark ID (A) registered in the mark ID storage location 261 of the collaboration group list 260, the collaboration unit 240 deletes data having the device ID 282 and address 283 included in the completion notification data 280b from the collaboration group list 260.

[0096] In this way, the collaboration unit 240 manages the HMDs 100 belonging to the collaboration group by transmitting and receiving the notification data 280 and updating the collaboration group list 260 accordingly. In this embodiment, the collaboration group list 260 is generated and managed in the same way in all HMDs 100 that detect the same collaboration mark 810 and execute the collaboration app. As a result, the collaboration group list 260 is shared among the HMDs 100.

[0097] The above functions are realized by the main processor 111 loading a program stored in the ROM 114 into the RAM 113 and executing it. The lists and data are stored in the storage device 110.

[0098] With the above configuration, the HMD 100 uses the outer camera 133 to capture the field of view in front of the HMD 100, and when a linking mark 810 such as an icon is captured within the captured field of view or within the linking field of view 131a, the application associated with the linking mark 810 can be executed in conjunction as a linking app.

[0099] [Linkage processing] Next, the flow of the cooperative processing of this embodiment will be described with reference to Fig. 6. Here, the processing in one HMD 100 will be described.

[0100] When the detection unit 220 detects the linkage mark 810 (step S1101), the detection unit 220 identifies an application registered in the application list 250 in association with the detected linkage mark 810 as a linkage application (step S1102).

[0101] The collaboration unit 240 generates the collaboration group list 260 (step S1103) and transmits the detection notification data 280a to the surrounding HMDs 100 by short-range wireless communication (step S1104). Note that either the generation of the collaboration group list 260 or the transmission of the detection notification data 280a may be performed first.

[0102] The execution unit 230 starts the linked application and executes the linked application until an instruction to terminate the application is received (steps S1105 and S1106).

[0103] When detection notification data 280a is received from a nearby HMD 100 during execution of the linked app (step S1107), if the detection notification data 280a has the same mark ID 281 as the linked mark 810 detected by the link unit 240 itself, the link unit 240 updates the linked group list 260 (step S1108). Here, the device ID 282 and address 283 of the detection notification data 280a are added to the linked group list 260. Thereafter, the linked app is executed based on the updated linked group list 260.

[0104] Furthermore, when an instruction to end the linked app is received, the execution unit 230 ends the linked app, and the link unit 240 transmits end notification data to the surrounding HMDs 100 (step S1109), and the process ends.

[0105] As described above, the information processing system 700 of this embodiment executes an application in cooperation with a plurality of HMDs 100, which are information processing terminals. Each HMD 100 includes a detection unit 220 that detects a predetermined coordination mark 810, and an execution unit 230 that executes an application associated with the detected coordination mark 810 as a coordination app in cooperation with other HMDs 100 that have detected the same coordination mark 810. Furthermore, each HMD 100 of this embodiment includes a coordination unit 240 that manages other HMDs 100 that have detected the same coordination mark 810 as constituent terminals (constituent members) of a coordination group that executes a coordination app in cooperation with each other.

[0106] As described above, according to this embodiment, by the user 900 recognizing the same coordination mark 810, it is possible to automatically identify the coordination app to be launched and executed on multiple HMDs 100 and to identify the HMDs 100 to be coordinated. That is, according to this embodiment, simply by detecting the same coordination mark 810 on each HMD 100, a coordination group can be formed and the same or equivalent coordination app can be executed in coordination. For example, by forming a group among users 900 wearing information terminals such as the HMD 100, it is possible to easily and conveniently chat, hold a meeting, play games, and the like among the group.

[0107] Furthermore, the HMD 100 that does not detect the coordination mark 810 does not join the coordination group. This prevents coordination with the HMD 100 of an unintended user 900, ensuring safety.

[0108] Conventionally, when an HMD 100 that was not a member of a collaboration group at the time of initial setup joins later, the collaboration group needs to be reconstructed. Reconstruction requires the members of the collaboration group to take the time to add the person joining later to the group. However, according to this embodiment, the latest members can be managed in the collaboration group list, so that the person joining later can join the collaboration group simply by looking at the collaboration mark 810. Therefore, the operations that were conventionally required are no longer necessary, and usability can be significantly improved.

[0109] As described above, according to this embodiment, application linkage can be realized between a plurality of information processing terminals in a simple and user-friendly manner.

[0110] When launching a linked app using the method of this embodiment, the HMD 100 to be linked may be limited to those that communicate only within a local network with a limited connection range, such as a local area network (LAN), personal area network (PAN), or virtual private network (VPN), for safety reasons.

[0111] <Variation 1> In the above embodiment, each HMD 100 identifies an associated app and then starts the associated app, but this is not limited to this. For example, the HMD 100 that first detects the association mark 810 may be set as a main device, and the main HMD 100 may transmit an instruction to start an associated app to the other HMDs 100, and the other HMDs 100 may start the associated app after receiving the instruction.

[0112] In this case, for example, the HMD 100 that detects the coordination mark 810 earliest is designated as the main device. In this case, if the coordination unit 240 has identified a coordinated app and has not yet received the detection notification data 280a, it transmits the detection notification data to the surrounding HMDs 100 and sets a flag designating the HMD 100 as the main device. Then, the coordination unit 240 of the main device generates a coordination group list 260 and registers the HMD 100 in the coordination group list 260 each time it receives detection notification data 280a from the surrounding HMDs 100 for a predetermined period of time. After the predetermined period has elapsed, the coordination unit 240 outputs an application launch instruction to the HMDs 100 registered in the coordination group list 260.

[0113] After detecting the coordination mark 810, the coordination unit 240 of an HMD 100 other than the main device receives, from a surrounding HMD 100, detection notification data 280a of the same mark ID (A) as the mark ID (A) corresponding to the coordination mark 810, and then generates detection notification data 280a of the own device as a reply and transmits it to the HMD 100 (main device) that sent it. After that, when an application start instruction is received, the execution unit 230 starts and executes the coordination application.

[0114] When a main device is provided, the cooperation group list 260 may be provided in all HMDs 100 in the cooperation group, or only in the main device. When only the main device has the cooperation group list 260, cooperation processing is performed via the main device when the cooperation app is executed.

[0115] Furthermore, the main device is not limited to the HMD 100 that first detected the coordination mark 810. For example, the main device may be determined in advance.

[0116] In this modification, when the main device transmits the detection notification data 280a, the HMD 100 that detects the same coordination mark 810 receives the detection notification data 280a in return, but the present invention is not limited to this. For example, the main device may be configured to inquire of all HMDs 100 within a predetermined search range as to whether or not they recognized the coordination mark 810. Then, the HMDs 100 that receive a response indicating detection are registered in the coordination group list 260.

[0117] In this case, the search time can be shortened by limiting the search range to a predetermined range. For example, the search range can be limited to devices within the aforementioned LAN, PAN, or VPN, HMDs 100 that can communicate directly via short-range wireless communication (Wi-Fi (registered trademark), Bluetooth, etc.), HMDs 100 connected to the same access point, or HMDs 100 with a specified number of network hops (the number of intermediate network devices that data must pass through between the source and destination). The search can also be configured to be terminated after a specified time. This will inevitably exclude HMDs 100 with a large number of hops from the search, thereby achieving the same purpose.

[0118] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, collaboration is achieved only by the HMD 100. Furthermore, except for modified examples, each individual HMD 100 basically manages a collaboration group. On the other hand, in this embodiment, a server, which is a second information processing terminal different from the HMD 100, manages the collaboration group and achieves collaboration. Hereinafter, this embodiment will be described, focusing on the configuration different from the first embodiment.

[0119] 7, an information processing system 701 of this embodiment further includes a server 400 in addition to the configuration of the information processing system 700 of the first embodiment. The multiple HMDs 100 can transmit and receive data to and from each other via short-range wireless communication. Each of the multiple HMDs 100 can also transmit and receive information to and from the server 400 via a network 300 such as a LAN. The HMDs 100 are connected to the network 300 via, for example, a wireless router or the like.

[0120] The hardware configuration of the HMD 100 of this embodiment is the same as that of the first embodiment. The functional configuration of the HMD 100 of this embodiment is also basically the same as that of the first embodiment. However, the configurations of the linking unit 240 and the execution unit 230 are different. The data stored in the storage device 110 is also different.

[0121] In this embodiment, similarly to the first embodiment, when the linking mark 810 is detected, the linking unit 240 transmits detection notification data 280a to the server 400. Furthermore, when the linking app is terminated, the linking unit 240 transmits termination notification data 280b to the server 400. Note that in this embodiment, the detection notification data 280a may also include current position information of the HMD 100.

[0122] In this embodiment, the server 400 manages a group of HMDs 100 that cooperate to execute a cooperative application as a cooperative group.

[0123] The server 400 may be of any form as long as it can be connected to each HMD 100 via communication and send and receive necessary information, and may be, for example, a local server, a cloud server, an edge server, or a network service.

[0124] 8(a) is a block diagram of functions related to the processing of the server 400 of this embodiment. As shown in this figure, the server 400 includes a main control unit 410, a transmission / reception unit 420, a cooperation control unit 430, and a cooperation management unit 440. The server 400 also manages a cooperation group list 460.

[0125] The transmitter / receiver 420 receives the notification data 280 from each HMD 100 .

[0126] The cooperation management unit 440 executes cooperation management processing to manage a group of HMDs 100 that cooperate to execute the same cooperation app as a cooperation group. In this embodiment, one cooperation group list 460 is generated for a group of HMDs 100 that detect the same cooperation mark 810, and the cooperation group is managed by this list.

[0127] An association group list 460 is generated for each mark ID. Each association group list 460 basically has the same configuration as the association group list 260 of the first embodiment. That is, as shown in FIG. 8(b), for each mark ID that identifies an association mark 810, the association group list 460 includes a mark ID storage location 461 that stores the mark ID, a device ID 462 of the HMD 100 that executes the association app in association with the mark ID, and an address 463. In this embodiment, when the association group list 460 is received from the server 400, the association group list 260 stored in the storage device 110 is replaced with the latest one.

[0128] When the execution of the linked app is performed via the server 400, the linkage control unit 430 controls the execution of the linked app among the multiple HMDs 100. The execution of the linked app is controlled based on the linkage group list 460. For example, when data is received, the linkage control unit 430 transmits the data to HMDs 100 other than the source HMD 100 in the linkage group that includes the device ID of the source HMD 100. The linkage control unit 430 is used in a modified example described later.

[0129] 8(c) is a hardware configuration diagram of the server 400. The server 400 of this embodiment includes a CPU 401, a memory 402, a storage device 403, and a communication interface 404. Each of the above functions is realized by the CPU 401 loading a program stored in the storage device 403 into the memory 402 and executing the program.

[0130] With the above configuration, in this embodiment as well, a collaboration group can be formed among a plurality of HMDs 100, and a common application can be executed in collaboration.

[0131] [Linkage processing] Next, the flow of processing when an integrated application is executed (integrated application execution processing) according to this embodiment will be described with reference to Fig. 9. Here, the description will be given taking as an example transmission and reception of data with one HMD 100. The same applies to transmission and reception with other HMDs 100.

[0132] When the detection unit 220 of the HMD 100 detects the coordination mark 810 (step S1101), the detection unit 220 identifies an application registered in association with the detected coordination mark 810 as a coordination application (step S1102).

[0133] The collaboration unit 240 of the HMD 100 generates the detection notification data 280a and transmits it to the server 400 (step S2104).

[0134] When the collaboration management unit 440 of the server 400 receives the detection notification data 280a (step S2201), the collaboration management unit 440 stores the data of the HMD 100 that sent the detection notification data 280a in the collaboration group list 460, thereby updating the collaboration group list 460 (step S2202). Here, the device ID 282 and the address 283 are added to the collaboration group list 460 in which the mark ID corresponding to the mark ID 281 in the detection notification data 280a is stored in the mark ID storage location 461. Note that if there is no collaboration group list 460 in which the mark ID corresponding to the mark ID 281 in the detection notification data 280a is stored in the mark ID storage location 461, a new one is created and then added.

[0135] Then, when the collaboration management unit 440 finishes updating, it returns data indicating that the process of joining the collaboration group has been completed and instructing the HMD 100 to start up (start-up instruction) to the HMD 100 that sent the data (step S2203). Also, it transmits the collaboration group list to all HMDs 100 registered in the collaboration group list.

[0136] When the execution unit 230 of the HMD 100 receives the start instruction, it starts and executes the linked app (step S1105). As in the first embodiment, the execution unit 230 of the HMD 100 continues to execute the executed app until it receives an instruction to end the executed app (step S1106).

[0137] During execution of the cooperative application, in this embodiment, the execution unit 230 transmits and receives data required for cooperative execution to and from other HMDs 100 registered in the cooperative group list (step S2108).

[0138] When the execution unit 230 of the HMD 100 receives an instruction to terminate the linked app from the user 900, it terminates the linked app, and the collaboration unit 240 generates termination notification data 280b, transmits it to the server 400 (step S2109), and terminates the process. Meanwhile, the collaboration management unit 440 of the server 400 that received the termination notification data 280b (step S2205) updates the linked group list 460 by deleting the HMD 100 that sent the data from the linked group list 460 (step S2206). Furthermore, the updated linked group list 460 is transmitted to each HMD 100 registered in the updated linked group list 460.

[0139] The same applies to the other HMDs 100; when the HMDs 100 detect the collaboration mark 810, they transmit the detection notification data 280a to the server 400. When the server 400 receives the detection notification data 280a or the end notification data 280b, it updates the collaboration group list 460 as needed and returns the updated collaboration group list 460.

[0140] As described above, the information processing system 701 of this embodiment executes an application in cooperation with a plurality of HMDs 100, which are information processing terminals. Each HMD 100 includes a detection unit 220 that detects a predetermined coordination mark 810 and an execution unit 230 that executes an application associated with the detected coordination mark 810 as a coordinated app in cooperation with other HMDs 100 that have detected the same coordination mark 810. The information processing system 701 of this embodiment further includes a server 400 that includes a coordination manager 440 that manages multiple HMDs 100 that have detected the same coordination mark 810 as members of a coordination group that coordinately executes a coordinated app. The coordination manager 440 transmits data obtained through a coordination management process to the other HMDs 100 and receives data obtained through the coordination management process from the other HMDs 100. Meanwhile, the execution unit 230 of the HMD 100 transmits data obtained through processing of the coordinated app to the other HMDs 100 and receives data obtained through processing of the coordinated app from the other HMDs 100.

[0141] According to this embodiment, similar to the first embodiment, application linkage can be realized simply, safely, and conveniently. Furthermore, according to this embodiment, the linked HMDs 100 are managed by the server 400. As long as the HMDs 100 can detect the same linkage mark 810 and can communicate with the server 400, there are no restrictions on the placement of the HMDs 100, and a system with a higher degree of freedom can be realized.

[0142] <Variation 2> In the HMD 100 of this embodiment, when a start instruction is received from the server 400, the linked app is started. However, this is not limited to this. For example, similar to the first embodiment, when a linked app is identified, the HMD 100 may start the linked app on its own. In this case, the start instruction does not need to be transmitted from the server 400.

[0143] <Variation 3> The activation instruction may also be transmitted simultaneously. For example, this may occur when multiple HMDs 100 visually recognize the coordination mark 810 and form a coordination group to execute a coordination app at approximately the same time. In this case, the server 400 simultaneously transmits participation acceptance data to multiple HMDs 100 that are the senders of the detection notification data 280a having the same mark ID 281 within a predetermined period of time.

[0144] <Variation 4> The server 400 manages each HMD 100 that detects the coordination mark 810 as a coordination group. However, the management of the coordination group is not limited to this.

[0145] For example, a group of HMDs 100 that have detected the same coordination mark 810 may be divided into multiple coordination groups and managed. In this case, the groups may be divided, for example, by the time when the detection notification data 280a was transmitted. Alternatively, the groups may be divided by the position information of the HMDs 100.

[0146] When dividing by time, for example, as described above, when the cooperation management unit 440 receives detection notification data 280a including the same mark ID 281 consecutively within a predetermined period, the cooperation management unit 440 groups the HMDs 100 that are the senders of the detection notification data 280a into one cooperation group. After that, after the predetermined period has elapsed, the group of HMDs 100 that have sent detection notification data 280a including the same mark ID 281 are grouped into another cooperation group.

[0147] 10, when multiple groups of HMDs 100 see the same collaboration mark 810 and start chatting at different times, the collaboration management unit 440 divides the HMDs 100 into groups based on the timing at which the chats start. Then, a collaboration group list 460 for each group is generated and managed. Here, a collaboration group list 460 for collaboration group 1 and a collaboration group list 460 for collaboration group 2 are generated and managed. Note that in this case as well, an HMD 104 for which the user 900 does not recognize the collaboration mark 810, i.e., which does not detect the collaboration mark 810, does not join any collaboration group.

[0148] When dividing the HMD 100 based on the position information of the HMD 100, the cooperation management unit 440 performs group division using the position information transmitted from the HMD 100. The boundaries of the group division are determined in advance.

[0149] <Variation 5> In the above embodiment, the linked app is executed between the HMDs 100 in the linked group, but this is not limiting. For example, the linked app may be configured to be executed via the server 400. In this case, the above-described link control unit 430 controls the execution of the linked app. Furthermore, the transmission / reception unit 420 mediates the transmission and reception of data between the HMDs 100 when multiple HMDs 100 execute the linked app in collaboration with each other.

[0150] In this case, the cooperation group list 460 may be managed only by the server 400. That is, the cooperation management unit 440 does not need to transmit the cooperation group list 460 to the HMD 100 in step S2203 or step S2206.

[0151] In each HMD 100, while a cooperative application is being executed, the execution unit 230 transmits data required for cooperative execution to the server 400. In addition, in the server 400, the cooperation control unit 430 refers to the cooperation group list 460, and when data is received from an HMD 100 in the cooperation group, the cooperation control unit 430 processes the data and transmits it to the other HMDs 100 in the cooperation group.

[0152] The server 400 may store applications, files, and data on the server 400 and transfer them via communication to each HMD 100 as needed. An application may be started on the server 400, and playback information such as video, screen display, and audio may be transferred to the HMDs 100 that are members of the collaboration group.

[0153] <<Third Embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the link mark to be recognized is displayed on the display of an information processing terminal such as a smartphone. In the following, this embodiment will be described assuming that the information processing terminal is a smartphone.

[0154] 11 is a diagram illustrating an overview of an information processing system 702 according to this embodiment. As shown in this diagram, the information processing system 702 according to this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a smartphone 500.

[0155] In this embodiment, each HMD 100 determines whether or not it is in a linked state with the smartphone 500. Then, the smartphone 500 manages the group of HMDs 100 that are in a linked state with itself as constituent members of a linked group.

[0156] An information processing system 702 of this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a smartphone 500. In this embodiment, the cooperation mark 820 is displayed on the display 531 of the smartphone 500.

[0157] The HMD 100 of this embodiment is basically the same as that of the first embodiment, except for the processing of the detection unit 220 and the linkage unit 240 and the data that is held.

[0158] The detection unit 220 of the HMD 100 in this embodiment first determines whether the smartphone 500 and the HMD 100 are in a predetermined positional relationship. That is, it determines whether the smartphone 500 is in a predetermined positional state in which it can operate in a linked operation mode, which will be described later. In this embodiment, as shown in FIG. 12(a), the detection unit 220 displays the linked field of view 131a on the display 131, analyzes the image captured by the outer camera 133, and determines whether the smartphone 500 is in the predetermined positional state based on the displayed content.

[0159] The associated field of view 131a is set to a partial area of ​​the field of view that the user 900 can see through the opening portion (lens portion) of the HMD 100 or the display area of ​​the display 131. Note that the entire field of view that the user 900 can see through the HMD 100 may be set as the associated field of view 131a by changing the device configuration, setting values, etc.

[0160] The detection unit 220 determines that the state is a predetermined arrangement state when a predetermined app icon 822 displayed on the display 531 of the smartphone 500 is within the linked field of view 131a, as shown in Fig. 12(a). Generally, a plurality of icons 822, 823, and 824 are displayed on the display 531 of the smartphone 500, as shown in Fig. 12(a). The predetermined icon is, for example, the icon of the app selected by the user 900.

[0161] Whether or not the icon 822 of the application selected by the user 900 is included in the linked visual field 131a is determined by analyzing the captured image 136. For example, the detection unit 220 determines the state and position of the pointing finger of the user 900 by image analysis processing.

[0162] When the detection unit 220 determines that the predetermined arrangement state is established, it detects the icon 822 of the predetermined application as the linkage mark 820. Then, it identifies the application registered in the application list 250 in association with the detected linkage mark 820 as the linked application. Note that if the application is not registered in the application list 250, it may be obtained from an external device such as a server and registered in the application list 250, as in the first embodiment.

[0163] In this embodiment, the application list 250 may store shape data of icons of applications installed on the device as the shape data 253 of the mark data 252.

[0164] Furthermore, as shown in Figure 12 (b), if the specified icon 822 displayed on the display 531 of the smartphone 500 is not positioned within the linked field of view 131a, the detection unit 220 determines that the HMD 100 and the smartphone 500 are not in the specified position, and does not perform any subsequent detection processing, etc.

[0165] When the linking unit 240 detects the linking mark 820 , the linking unit 240 generates detection notification data 280 a and transmits it to the smartphone 500 .

[0166] The execution unit 230 starts and executes the linked app when it receives a start instruction from the smartphone 500. Note that, also in the present embodiment, when a linked app is identified, the linked app may be started in the HMD 100 without a start instruction from the smartphone 500.

[0167] Next, the smartphone 500 of this embodiment will be described.

[0168] FIG. 13 is a hardware configuration diagram of a smartphone 500. As shown in this diagram, the smartphone 500 includes a main processor 511, a storage device 510, an operation reception device 520, an image processing device 530, a sound processing device 540, a sensor 550, a communication device 560, an expansion interface unit 591, a timer 592, a vibration generator 593, and an internal bus 512 connecting these. The storage device 510 also includes a RAM 513, a ROM 514, and a flash memory 515. The operation reception device 520 includes operation keys 521. The image processing device 530 includes a display 531, an outer camera 533, and an inner camera 534. The sound processing device 540 includes a speaker 541 and a microphone 543. The sensor 550 includes a GPS receiver 551, a gyro sensor 552, a geomagnetic sensor 553, an acceleration sensor 554, and a depth sensor 555. The communication device 560 includes a LAN communication interface 561, a telephone network communication interface 562, and a short-range wireless communication interface 563. These have the same functions as the components of the same names in the HMD 100.

[0169] However, the smartphone 500 has a display 531 with a larger area than the HMD 100. For this reason, the operation reception device 520 further includes a touch panel 523 having a touch sensor. For example, a touchpad-type input means such as a capacitance type may be included. In this case, approach or contact operation (touch operation) using a finger, a touch pen, or the like is detected as an operation input. The touchpad type allows the user 900 to set and input information that he or she wishes to input, or to easily select and specify app or file icons on the display 531 by touch operation. The touch panel 523 is disposed over the display 531. Note that instruction input may also be received via a keyboard or the like connected to the expansion interface unit 591.

[0170] The display 531 is configured with a liquid crystal panel or the like, and displays notification information to the user 900, such as the remaining battery capacity, various alarms, and the time, as well as the icons of apps to be launched and the execution status of the apps within the display 531. The operation reception device 520 is, for example, a touchpad-type input means such as a capacitance type, and detects approach or contact operations (touch operations) with a finger, a touch pen, or the like as operation input. The user 900 sets and inputs the information he or she wants to input, and easily selects and specifies app or file icons on the display 531 by touch operations.

[0171] The smartphone 500 of this embodiment displays an icon for accepting an instruction to start an app on the display 531. This icon is a mark for accepting an instruction to specify and start an app by approaching or contacting the icon with a finger, a touch pen, or the like of the user 900 (hereinafter referred to as "touch").

[0172] Next, a description will be given of the functions of the smartphone 500 of this embodiment. FIG.

[0173] As shown in this figure, the smartphone 500 of this embodiment includes a main control unit 570, a reception unit 571, a transmission / reception unit 572, a collaboration management unit 573, an application execution unit 574, an output unit 575, an icon data storage unit 577, an application program storage unit 576, and a collaboration group list 578.

[0174] The main control unit 570 controls the operation of each unit of the smartphone 500. In this embodiment, the main control unit 570 also determines the operation mode of the smartphone 500. In this embodiment, the main control unit 570 has two operation modes: a standalone operation mode in which the smartphone 500 executes an app by itself, and a cooperative operation mode in which the smartphone 500 executes an app in cooperation with the HMD 100.

[0175] For example, when the main control unit 570 receives the detection notification data 280a from the HMD 100 with which communication has been established, the main control unit 570 determines the operation mode to be the cooperative operation mode.

[0176] The reception unit 571 receives instructions from the user 900. In this embodiment, this is realized by the main processor 511 operating the operation reception device 520. In this embodiment, for example, various operation instructions such as an instruction to start an application are received.

[0177] The transmission / reception unit 572 transmits and receives data to and from an external device. In this embodiment, this is realized by the main processor 511 operating the communication device 560. In this embodiment, data is transmitted and received to and from the HMD 100 via the transmission / reception unit 572.

[0178] The collaboration management unit 573 performs collaboration when the operation mode of the smartphone 500 is determined to be the collaboration operation mode. The collaboration management unit 573 registers the mark ID 281, the device ID 282, and the address 283 of the received detection notification data 280a in the collaboration group list 578. Furthermore, when the collaboration management unit 573 receives termination notification data 280b from an HMD 100 registered in the collaboration group list 578, it deletes the data of the corresponding device. In this way, the collaboration group is managed. The configuration of the collaboration group list 578 is the same as the collaboration group list 260 in the first embodiment.

[0179] The collaboration management unit 573 also controls the collaboration of collaboration apps by the HMDs 100 that are members of each collaboration group. For example, it displays an in-use mark on the icon of an app that is being used, or displays the number of participants, making it easy to see which apps are running. This allows people who join later to check the apps that other participants are using on the display and then select and join.

[0180] In the standalone operation mode, the application execution unit 574 executes an application on the smartphone 500. The standalone operation mode is an operation mode in which, upon receiving an instruction to start an application from the user 900, the smartphone 500 starts and executes the application. The program of the application to be executed is stored in the application program storage unit 576.

[0181] The output unit 575 outputs the execution result of the application executed by the application execution unit 574. In this embodiment, the execution result is output to the display 531 and the speaker 541, for example.

[0182] Application programs are stored in application program storage unit 576. Icons for receiving instructions to execute each application program are stored in icon data storage unit 577. The icons are stored in association with the applications.

[0183] The above functions are realized by main processor 511 loading a program stored in ROM 514 into RAM 513 and executing it. An application program storage unit 576 and an icon data storage unit 577 are stored in storage device 510.

[0184] [Linkage processing] The flow of the linking process of this embodiment will be described with reference to FIG.

[0185] First, communication is established between the HMD 100 and the smartphone 500 (steps S3101 and S3201). Here, communication settings are made between the smartphone 500 and multiple HMDs 100 to enable communication. For example, when the HMD 100 enters the communication range of the smartphone 500, communication is established between the two automatically or manually.

[0186] After that, the smartphone 500 displays the icon 822 (step S3202). Note that here, a normal screen displaying a large number of icons is displayed. The icon 822 to be used as the linkage mark 820 is selected and instructed by the user 900 as described above.

[0187] The subsequent processing is substantially the same as the collaboration processing with the server 400 in the second embodiment. That is, when the detection unit 220 of the HMD 100 detects the collaboration mark 820 (step S1101), it identifies an app registered in association with the detected collaboration mark 820 as a collaboration app (step S1102).

[0188] The link unit 240 of the HMD 100 generates the detection notification data 280a and transmits it to the smartphone 500 (step S2104).

[0189] When the cooperation management unit 573 of the smartphone 500 receives the detection notification data 280a (step S3203), the cooperation management unit 573 stores the data of the HMD 100 that is the sender in the cooperation group list 578, thereby updating the cooperation group list 578 (step S3204).

[0190] Then, when the update is completed, the cooperation management unit 573 returns data indicating that the process of joining the cooperation group has been completed and instructing the HMD 100 to start up (start-up instruction) to the transmission source HMD 100 (step S3205). In addition, the cooperation group list is transmitted to all HMDs 100 registered in the cooperation group list 578.

[0191] When the execution unit 230 of the HMD 100 receives the start instruction, it starts and executes the linked app (step S1105). As in the first embodiment, the execution unit 230 of the HMD 100 continues to execute the executed app until it receives an instruction to end the executed app (step S1106).

[0192] During execution of the cooperative application, in this embodiment, the execution unit 230 transmits and receives data required for cooperative execution to and from other HMDs 100 registered in the cooperative group list 578 (step S2108).

[0193] When the execution unit 230 of the HMD 100 receives an instruction to terminate the linked app from the user 900, the execution unit 230 terminates the linked app, and the collaboration unit 240 generates and transmits termination notification data 280b to the smartphone 500 (step S2109), and ends the processing. Meanwhile, the collaboration management unit 573 of the smartphone 500 that received the termination notification data 280b (step S3207) updates the linked group list 578 by deleting the HMD 100 that sent the data from the linked group list 578 (step S3208). Furthermore, the updated linked group list 578 is transmitted to each HMD 100 registered in the updated linked group list 578.

[0194] The same applies to other HMDs 100, and when the HMD 100 detects the collaboration mark 810, it transmits detection notification data 280a to the smartphone 500. When the smartphone 500 receives the detection notification data 280a or the end notification data 280b, it updates the collaboration group list 578 as needed and returns the updated collaboration group list 578.

[0195] In this embodiment, if there is a time lag between determining whether the predetermined arrangement state is established and the user 900 touching the icon 822, an HMD 100 that is not looking at the icon 822 at that time may join the collaboration group, or an HMD 100 that is looking at the icon 822 at that time may not join the collaboration group. For this reason, the storage interval (the interval at which the collaboration group list 578 is updated) may be shortened so that a time lag does not occur between the two. Also, the icon touch may be recognized by the outer camera 133 of the HMD 100. In this case, the determination of the time of the touch moment may vary depending on the viewing angle, so the determination period for the icon touch may be set to a certain time range.

[0196] As described above, the information processing system 702 of this embodiment executes an application in cooperation with a plurality of HMDs 100, which are information processing terminals. Each HMD 100 includes a detection unit 220 that detects a cooperation mark 820 displayed on the display 531 of the smartphone 500, and an execution unit 230 that executes an application associated with the detected cooperation mark 820 as a cooperation app in cooperation with other HMDs 100 that have detected the same cooperation mark 820. Furthermore, the information processing system 702 of this embodiment further includes a smartphone 500 that includes a cooperation management unit 573 that manages multiple HMDs 100 that have detected the same cooperation mark 820 as members of a cooperation group that cooperates to execute a cooperation app, and the execution unit 230 transmits data obtained by processing the cooperation app to the other HMDs 100 and receives data obtained by processing the cooperation app from the other HMDs 100.

[0197] According to this embodiment, similar to the first embodiment, application linkage can be achieved simply, safely, and conveniently. Furthermore, according to this embodiment, a linkage mark 820 is displayed on the display 531 of the smartphone 500. Therefore, when the HMD 100 and the smartphone 500 are in a predetermined positional state, such as when the icon 822 displayed on the display 531 of the smartphone 500 is visible within the linkage field of view 131a of the HMD 100, a linkage group can be formed between HMDs 100 that are in the same positional state as the smartphone 500, and linked apps can be executed in a linked manner. Therefore, according to this embodiment, it is possible to simply and conveniently identify linked apps and identify linked HMDs 100.

[0198] Furthermore, according to this embodiment, the coordination mark 820 is displayed on the display 531 of the smartphone 500. Unlike coordination marks that are physically printed and cannot be easily changed, this embodiment allows a large number of coordination marks to be displayed within a limited area while being swapped by scrolling or the like. This increases the number and options of usable coordination marks. In addition, the placement position of the coordination mark 820 can be moved. This allows the user 900 of the HMD 100 to recognize the coordination mark 820 in a position that is easy to see, making it easy for the user 900 to visually recognize the coordination mark 820. In other words, it becomes easier for the HMD 100 to detect the coordination mark 820.

[0199] Furthermore, the smartphone 500 can distinguish between the cooperative operation mode and the independent operation mode. Therefore, when the user 900 selects the icon 822 displayed on the display 531, it can automatically determine whether to run the corresponding app on the smartphone 500 or to control the cooperative operation.

[0200] <Variation 6> In the above embodiment, the icon 822 of the smartphone 500 is viewed through the field of view of multiple HMDs 100. For this reason, when multiple icons are displayed on the display 531 of the smartphone 500, the user 900 needs to keep the icon 822 selected at all times while the HMD 100 with which the user wishes to link detects the icon 822 as the link mark 820. However, the present invention is not limited to this method.

[0201] For example, when the smartphone 500 first receives the detection notification data 280a and generates the collaboration group list 578, it may display only the icon 822 selected by the user 900 on the display 531. After that, the icon 822 functions as the collaboration mark 820 without any selection instruction from the user 900.

[0202] As a result, a specific icon 822 is displayed on the display 531 as the collaboration mark 820 for a predetermined period of time, making it easier for the HMD 100 that wants to join the collaboration group to do so. On the other hand, even with this configuration, the user 900 who does not want to participate can simply not look at the display of the smartphone 500.

[0203] <Variation 7> In the above embodiment, the smartphone 500 receives the detection notification data 280a and transmits the activation instruction to the HMD 100 that has transmitted the detection notification data 280a each time the smartphone 500 receives the detection notification data 280a and registers the data in the collaboration group list 578. However, this is not limited to this. For example, the smartphone 500 may transmit the activation instruction to all the HMDs 100 that have transmitted the detection notification data 280a.

[0204] In this case, for example, the cooperation management unit 573 registers each received detection notification data 280a in the cooperation group list 578. After that, when a start instruction is received by touching the icon 822 set as the cooperation mark 820, the cooperation management unit 573 transmits the start instruction to all HMDs 100 registered in the cooperation group list 578. When each HMD 100 receives the start instruction from the smartphone 500, it starts the cooperation app in response to the start instruction.

[0205] <Variation 8> Also in this embodiment, similarly to the modified example of the second embodiment, the smartphone 500 may control the execution of the linked app. In this case, the link management unit 573 transmits data received from each HMD 100 that executes the linked app to the other HMDs 100.

[0206] In this case, the cooperation group list 578 may be managed only in the smartphone 500. That is, the cooperation management unit 573 does not need to transmit the cooperation group list 578 to the HMD 100 in step S3205 or step S3208.

[0207] In each HMD 100, while the linked app is being executed, the execution unit 230 transmits data required for the linked execution to the smartphone 500. In addition, in the smartphone 500, the linkage management unit 573 refers to the linkage group list 578, and when it receives data from an HMD 100 in the linkage group, it processes the data and transmits it to the other HMDs 100 in the linkage group.

[0208] <Variation 9> Furthermore, the smartphone 500 may not only mediate the data of the linked app but also execute the linked app itself. In this case, the smartphone 500 registers its own device ID and address in the linked group list 578. Then, the application execution unit 574 executes the linked app even in the linked operation mode.

[0209] <Modification 10> In the above embodiment, whether or not the smartphone 500 and the HMD 100 are in a predetermined arrangement state is determined using an image captured by the outer camera 133 of the HMD 100. However, this is not limiting. For example, information from the sensor 550 of the smartphone 500 may be used to determine whether or not the smartphone 500 and the HMD 100 are in a predetermined arrangement state.

[0210] For example, the attitude and position of the HMD 100 are detected using an acceleration sensor 154, a gyro sensor 152, a geomagnetic sensor 153, etc., and the relationship between the attitude of the HMD 100 and the attitude and position of the smartphone 500 is determined. The attitude and position of the smartphone 500 are calculated using an acceleration sensor 554, a gyro sensor 552, and a geomagnetic sensor 553 provided in the smartphone 500. The distance between the smartphone 500 and the HMD 100 is acquired using a depth sensor 155, etc.

[0211] <Variation 11> In the above embodiment, when an icon 822 selected by the user 900 is present among the icons 822 displayed on the display 531 of the smartphone 500 or the like within the predetermined linkage field 131a, the predetermined arrangement state is determined, and the icon 822 is used as the linkage mark 820. However, this is not limiting. For example, the icon 822 to be used as the linkage mark 820 may be selected via the HMD 100 from among multiple icons displayed on the display 531 of the smartphone 500.

[0212] <Modification 12> Furthermore, the link mark 820 displayed on the display 531 of the smartphone 500 does not have to be the icon 822 of the link app. For example, it may be an icon indicating file data. In this case, the app that plays the file data is stored in association with the mark data as the app list 250. This allows the app to be launched on multiple HMDs 100, allowing the file data to be shared.

[0213] Note that file data may be playable not only by the same app but also by different apps. Therefore, different apps may be associated with the app list 250 in each HMD 100. This allows file data to be played back in a coordinated manner even between HMDs 100 equipped with different apps.

[0214] Furthermore, if the file data is an AR object, the coordinates of the AR object may be shared among the HMDs 100 by aligning the coordinate systems of the HMDs 100 with the linkage mark 820 as a reference.

[0215] <Variation 13> Furthermore, a specific icon 822 displayed on the display 531 of the smartphone 500 may be displayed in a manner that makes it clear (association display) while it is being used as the collaboration mark 820. Being used as the collaboration mark 820 means that the collaboration app associated with the icon 822 is being executed by multiple HMDs 100 in collaboration with each other.

[0216] The link display may be, for example, by changing the display color of the icon 822, blinking it, adding a small mark such as a dot 825 to the icon 822 as shown in Fig. 14(b), displaying the number of linked HMDs, etc. This allows the user to know which icon corresponds to which linked app is being executed in linkage, even when icons for multiple apps are displayed.

[0217] By displaying icons in this manner, it becomes easier to distinguish which apps are being executed in cooperation while the linked apps are being executed in cooperation, making it easier for a late joiner to distinguish which icons they should recognize.

[0218] <Variation 14> In each of the above embodiments and modifications, when a change occurs in the constituent members of a collaboration group, such as when a member is registered in the collaboration group list or, conversely, deleted from the collaboration group list, each HMD 100 may be configured to notify the user 900. The notification is performed, for example, by displaying on the display 131, outputting sound from the speaker 141, vibrating the vibration generator 173, or the like. This can improve convenience for the user 900.

[0219] <Variation 15> Furthermore, when another HMD 100 joins the collaboration group after the collaboration app has been linked and execution has begun, the HMD 100 may be configured to join with permission from all HMDs 100 already participating in the collaboration group or from the representative HMD 100.

[0220] For example, when an HMD 100 that wishes to participate transmits the detection notification data 280a, each HMD 100 or the representative HMD 100 checks whether participation is permitted. For example, this is checked by authentication processing. Then, only if permitted, the HMD 100 is added to the linked group list. Then, a notification of participation permission is sent to the HMD 100 that sent the data.

[0221] <Variation 16> The information terminal that executes the linked app in cooperation with the HMD 100 is not limited to the HMD 100. For example, it may be a smartphone 500, a personal computer, a tablet, or the like. A mixture of these information terminals may also be used. It is sufficient that the information processing terminal has at least a configuration that can determine that the link marks 810 and 820 have been detected and a configuration that can execute the linked app.

[0222] <Variation 17> Furthermore, the coordination mark 820 does not have to be a mark in the real space. For example, it may be a mark in a virtual reality (VR) space. For example, in the VR space, when avatars representing the respective users 900 recognize the same coordination mark, the avatars coordinate to execute the coordinated apps associated with the recognized coordination marks. Recognition in the VR space is realized, for example, by having the avatars touch (select) the coordination mark in the VR space. As a result, even when many users (avatars) exist in the VR space, the apps can be coordinated and executed only by the avatars of some of the users who have selected the coordination mark.

[0223] <Application example> In the third embodiment, the device that displays the linkage mark 820 is not limited to the display 531 of the smartphone 500. For example, it may be a screen that projects an image.

[0224] For example, all users 900 gathered in a conference room wear HMDs 100 and view a collaboration mark displayed on a screen (display) in the conference room through their field of vision. The HMDs 100 detect the collaboration mark. Each HMD 100 is associated with a collaboration app. Each HMD 100 identifies and launches the corresponding collaboration app. Launch notification data is sent and received, and collaboration groups are managed.

[0225] As a result, a collaboration group for the conference is automatically formed by the HMDs 100 of the users 900 who have gathered in the conference room, and collaboration applications can be executed in collaboration.

[0226] For example, by using a collaborative app that has various meeting processing functions, such as sharing documents only within the collaborative meeting group, recording participants' comments, and creating minutes, you can make meetings run more smoothly and easily increase efficiency.

[0227] Furthermore, by displaying a collaboration mark on one side of the screen during a meeting, an HMD 100 joining midway can simply detect the collaboration mark and launch the collaboration app to participate in the meeting. Particularly in meetings, the participating members often change each time. However, even in such a situation, according to this example, simply displaying the collaboration app icon on the screen can automatically form a collaboration group, allowing the meeting to proceed smoothly. This further improves convenience.

[0228] The second embodiment can also be applied to remote conferences. For example, when there are multiple conference locations. In this case, the same collaboration mark 810 is provided in advance at each conference location. The HMD 100 wishing to participate can automatically start a collaboration app by simply detecting the collaboration mark 810, and a collaboration group can be formed between the HMDs 100 in remote locations.

[0229] Interlocking applications can be executed in cooperation between the HMDs 100 registered as a cooperation group by communication over a network such as a LAN via the server 400. By making the interlocking applications applications that realize various functions of a conference, remote conferences can be easily carried out.

[0230] In this case, the linked app may be started simultaneously for each conference location or each icon installation location by a trigger such as touching an icon, or may be started at each timing of each HMD 100. After forming a linkage group for each conference location, the server 400 may integrate the linkage groups of all conference locations. In this case, the HMD 100 may be configured to connect to the server 400 via a representative HMD 100 of the linkage group of each conference location.

[0231] The present invention is not limited to the above-described embodiments and modifications, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0232] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0233] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0234] 100: HMD, 101: HMD, 102: HMD, 103: HMD, 104: HMD, 110: storage device, 111: main processor, 112: system bus, 113: RAM, 114: ROM, 115: flash memory, 120: operation reception device, 121: operation key, 130: image processing device, 131: display, 131a: linked field of view, 133: outer camera, 134: inner camera, 136: captured image ,137: Area, 140: Audio processing device, 141: Speaker, 143: Microphone, 150: Sensor, 151: GPS receiving unit, 152: Gyro sensor, 153: Geomagnetic sensor, 154: Acceleration sensor, 155: Depth sensor, 160: Communication device, 161: LAN communication I / F, 162: Telephone network communication I / F, 163: Short-range wireless communication I / F, 171: Expansion interface unit, 172: Timer, 173: Vibration generator, 210: Main control unit, 220: Detection unit, 230: Execution unit, 240: Collaboration unit, 250: Application list, 251: Application ID, 252: Mark data, 253: Shape data, 254: Mark ID, 260: Collaboration group list, 261: Mark ID storage location, 262: Device ID, 263: Address, 270: Processing data, 272: Mark data, 280: Notification data, 280a: Detection notification data, 280b: End notification data, 281: Mark ID, 282: Device ID, 283: Address, 284: Application ID, 285: Status, 300: Network, 400: Server, 401: CPU, 402: Memory, 403: Storage device, 404: Communication interface, 410: Main control unit, 420: Transmitter / receiver, 430: Collaboration control unit, 440: Collaboration management unit, 460: Collaboration group list, 461: Mark ID storage location, 462: Device ID, 463: Address, 500: Smartphone, 510: Storage device, 511: Main processor, 512: Internal bus, 513: RAM, 514: ROM, 515: Flash memory, 520: Operation reception device, 521: Operation keys, 523: Touch panel, 530: Image processing device, 531: Display, 533: Outer camera, 534: Inner camera, 540: Audio processing device, 541: Speaker, 543: Microphone, 550: Sensor, 551: GPS receiver, 552: Gyro sensor, 553: Geomagnetic sensor, 554: Acceleration sensor, 555: depth sensor, 560: communication device, 561: LAN communication interface, 562: telephone network communication interface, 563: short-range wireless communication interface, 570: main control unit, 571: reception unit, 572: transmission / reception unit, 573: collaboration management unit, 574: application execution unit, 575: output unit, 576: application program storage unit, 577: icon data storage unit, 578: collaboration group list, 591: expansion interface unit, 592: timer, 593: vibration generator, 700: Information processing system, 701: Information processing system, 702: Information processing system, 810: Collaboration mark, 820: Collaboration mark, 822: Icon, 823: Icon, 824: Icon, 825: Dot, 900: User, 901: User, 902: User, 903: User, 904: User, 912: Wireless communication, 913: Wireless communication, 914: Wireless communication, 923: Wireless communication, 924: Wireless communication, 934: Wireless communication, 941: Line of sight, 942: Line of sight, 943: Line of sight, 944: Line of sight

Claims

1. An information processing terminal that executes an application, a communication unit for transmitting and receiving information to and from other terminals; a detection unit that detects a predetermined linkage mark; a linking unit that, when the detection unit detects the linkage mark, transmits first notification data to the other terminal via the communication unit, the first notification data including information specifying that the linkage mark has been detected and first terminal identification information identifying the information processing terminal, and, when the communication unit receives second notification data from the other terminal, the second notification data including information specifying that the other terminal has detected the linkage mark and second terminal identification information identifying the other terminal, manages the other terminal as a member terminal of a linkage group that executes an application in cooperation with the information processing terminal; an execution unit that starts an application associated with the coordination mark, transmits data obtained by processing the started application to the other terminals managed by the coordination unit as constituent terminals of the coordination group, and receives data related to the processing of the application associated with the coordination mark from the other terminals, thereby executing the application associated with the coordination mark in coordination with the other terminals; The information processing terminal, wherein the detection unit detects a mark of a predetermined application as the link mark when it is determined that the information processing terminal is in a predetermined arrangement state.

2. 2. The information processing terminal according to claim 1, the information processing terminal further includes a camera that captures an image of the surroundings of the information processing terminal; The detection unit detects the surroundings of the information processing terminal based on the image of the surroundings acquired by the camera. and determining whether the terminal is in the predetermined arrangement state.

3. 2. The information processing terminal according to claim 1, the information processing terminal further includes a camera that captures an image of the surroundings of the information processing terminal; The detection unit detects the predetermined An information processing terminal that detects a linkage mark.

4. 2. The information processing terminal according to claim 1, The information processing terminal further includes a reception unit that receives an instruction from a user of the information processing terminal. Preparation, After the application associated with the link mark is executed, the accepting unit When an instruction to terminate the application is received, the execution unit terminates the execution of the application. and the communication unit transmits a completion notification to the other terminal.

5. 2. The information processing terminal according to claim 1, The information processing terminal receives the linkage mark and the application associated with the linkage mark. a memory for storing a list indicating the selection; the communication unit transmits and receives information to and from a server on a network; When the detection unit detects a mark that is not stored in the list, the communication unit 2. An information processing terminal that transmits inquiry data to the server by

6. 6. The information processing terminal according to claim 1, The other terminals are terminals located in the vicinity of the information processing terminal.

7. 6. The information processing terminal according to claim 1, The link mark is an icon for calling up the application's launch function. An information processing terminal characterized by the above.

Images