Information processing terminal

The described technology allows for easy and secure collaboration of applications among multiple information processing terminals by detecting cooperation marks, automatically forming groups, and executing shared applications, addressing the challenge of user-friendly collaboration.

JP2024164189AActive Publication Date: 2024-11-26MAXELL LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024144174
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-26
Estimated Expiration
2040-07-02

AI Technical Summary

Technical Problem

Existing technologies struggle to facilitate easy and user-friendly collaboration of applications between multiple information processing terminals, such as head-mounted displays (HMDs), as they do not effectively address the selection and initiation of cooperative communication among users.

Method used

An information processing terminal equipped with a communication unit, detection unit, and collaboration unit that detects a predetermined cooperation mark, allowing for automatic identification and execution of applications among terminals in a predetermined arrangement, forming a collaboration group without manual selection.

Benefits of technology

Enables convenient and secure collaboration of applications among multiple terminals by automatically forming groups based on detected cooperation marks, ensuring only intended participants are included and simplifying the process of starting shared applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164189000001_ABST
    Figure 2024164189000001_ABST
Patent Text Reader

Abstract

To easily and conveniently establish application coordination among a plurality of information processing terminals.SOLUTION: An information processing terminal is provided with: a communication unit for transmitting and receiving information to / from other terminals; a detection unit for detecting a predetermined cooperation mark; a cooperation unit for, when transmitting a notification indicating that the cooperation mark has been detected to the other terminal upon detecting the cooperation mark and receiving the notification indicating the detection of the cooperation mark from the other terminal, performing management to execute an application in cooperation with the other terminal as a constituent terminal of a cooperation group; and an execution unit for executing the application associated with the cooperation mark in cooperation with the other terminal. The detection unit detects a mark of a predetermined application as the cooperation mark when the information processing terminal determines that it is in a predetermined arrangement state.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for supporting the cooperative execution of applications (hereinafter referred to as apps) among a plurality of information processing terminals. [Background technology]

[0002] It is becoming common to execute applications in cooperation between multiple information terminals. For example, Patent Document 1 discloses an information processing system as a technology for selecting an optimal cooperation destination device to cooperate with a cooperation source device, which includes "a cooperation source device, multiple cooperation destination devices connectable to the cooperation source device, and an application cooperation device that controls cooperation of applications registered in the cooperation source device and the multiple cooperation destination devices, and the application cooperation device includes a first management unit that manages device information including information used to connect to the cooperation destination device and information regarding data used by the application registered in the cooperation destination device, a second management unit that refers to the device information in response to a cooperation request from the cooperation source device, and acquires cooperation candidates indicating cooperation destination devices and applications that can be cooperated with, and a cooperation control unit that starts the application indicated in the cooperation candidate and determines the device to be cooperated with and the application registered in the device that can be cooperated with as the cooperation destination (abstract excerpt)."

[0003] There is also a technique for controlling multiple devices in cooperation with each other based on a determination of the identity of images taken by the multiple devices. For example, Patent Document 2 discloses a determination device including "a first acquisition unit that acquires a first image taken by a first device together with the shooting time, a second acquisition unit that acquires a second image taken by a second device together with the shooting time, a determination unit that determines the identity of the shooting positions of the first image and the second image based on the respective captured images and shooting times of the first image and the second image, and an output unit that outputs the determination result to the first device or the second device (abstract excerpt)." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-142693 A [Patent Document 2] JP 2016-174253 A Summary of the Invention [Problem to be solved by the invention]

[0005] 2. Description of the Related Art Users wearing information terminals such as head mounted displays (HMDs) are increasingly forming groups and sharing and executing applications installed on the information terminals.

[0006] Although Patent Document 1 describes how to establish linked communication and launch an application on all information terminals on which the application can be launched, it does not suggest anything about selecting an information terminal to launch the application. For this reason, it is difficult to form a group among users and share an application.

[0007] Furthermore, although Patent Document 2 describes linking devices together based on the identity of captured images, it does not disclose anything about app launch or app linking.

[0008] The present invention has been made in view of the above circumstances, and has an object to provide a technique for achieving simple and easy-to-use linkage of applications between a plurality of information processing terminals. [Means for solving the problem]

[0009] The present invention is an information processing terminal that executes an application, comprising: a communication unit that transmits and receives information with other terminals; a detection unit that detects a predetermined coordination mark; a coordination unit that, when the detection unit detects the coordination mark, sends a notification to the other terminal indicating that the coordination mark has been detected by the communication unit, and when the communication unit receives a notification from the other terminal indicating that the coordination mark has been detected, manages the other terminal as a constituent terminal of a coordination group that executes an application in coordination with the other terminal; and an execution unit that executes an application associated with the coordination mark in coordination with the other terminal managed by the coordination unit as a constituent terminal of the coordination group, wherein the detection unit detects a mark of a predetermined application as the coordination mark when it determines that the information processing terminal is in a predetermined arrangement state. Effect of the Invention

[0010] According to the present invention, it is possible to realize simple and convenient application linkage between a plurality of information processing terminals. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiment. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of an information processing system according to a first embodiment; [Diagram 2] FIG. 1 is a configuration diagram of an information processing system according to a first embodiment. [Diagram 3] FIG. 2 is a hardware configuration diagram of the HMD according to the first embodiment. [Figure 4] FIG. 1A is a functional block diagram of an HMD according to a first embodiment, FIG. 1B is an explanatory diagram for explaining an example of an application list according to the first embodiment, FIG. 1C is an explanatory diagram for explaining an example of notification data according to the first embodiment, and FIG. 1D is an explanatory diagram for explaining an example of an association group list according to the first embodiment. [Diagram 5] 5A to 5C are explanatory diagrams illustrating detection of a linkage mark according to the first embodiment. [Figure 6]11 is a flowchart of a linking process according to the first embodiment. [Figure 7] FIG. 11 is a configuration diagram of an information processing system according to a second embodiment. [Figure 8] FIG. 1A is a functional block diagram of a server in a second embodiment, FIG. 1B is an explanatory diagram for explaining an example of an association group list in the second embodiment, and FIG. 1C is a hardware configuration diagram of a server in the second embodiment. [Figure 9] 13 is a flowchart of a linking process according to a second embodiment. [Figure 10] FIG. 11 is a system configuration diagram of a modified example of the second embodiment. [Figure 11] FIG. 13 is an explanatory diagram for explaining a system configuration of a third embodiment. [Figure 12] 13(a) and (b) are explanatory diagrams illustrating detection of a linkage mark by a detection unit according to a third embodiment. [Figure 13] FIG. 13 is a hardware configuration diagram of a smartphone according to a third embodiment. [Figure 14] FIG. 13A is a functional block diagram of a smartphone according to a third embodiment, and FIG. 13B is an explanatory diagram for explaining an example of icon display in a modified example of the third embodiment. [Figure 15] 13 is a flowchart of a linking process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <<First embodiment>> DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will now be described with reference to the drawings. Fig. 1 is an explanatory diagram for explaining an overview of an information processing system according to this embodiment.

[0013] In recent years, Virtual Reality (VR) technology, which creates a virtual world similar to reality and allows users to experience the sensation of being there, and Augmented Reality (AR) technology, which adds digital information to the real world and reflects and extends virtual spaces (virtual objects) created with CG (Computer Graphics) in the real world, have been used in games and various simulations. Head Mounted Displays (HMDs), which are equipped with displays and cameras, are widely used as information processing terminals that embody these technologies. Users wearing information processing terminals such as HMDs form groups, collaborate to run apps installed on the information terminals, and chat, hold meetings, play games, and so on among the groups.

[0014] It is desirable to execute these applications in a linked manner easily and safely, and this embodiment achieves this.

[0015] In this embodiment, multiple users 900, each wearing a head mounted display (HMD) 100, cooperate to execute a common application (app) installed in each HMD 100. The apps installed in each HMD 100 are the same or similar apps used for specific tasks such as chat, playing music and images, and creating documents.

[0016] The linkage is realized by the user 900 recognizing a mark corresponding to the application to be executed. The mark is, for example, a figure for calling a function such as starting an application, and may be printed or written somewhere, or may be displayed on the screen of an information processing terminal or the like. The mark is not limited to a geometric figure, and may be a character or the like. The mark may be any mark that can determine the application to be started, and is hereinafter referred to as a linkage mark 810. Furthermore, the application started on each HMD 100 by recognizing the linkage mark 810 does not need to be the same application. For example, the application started when a chat mark is recognized may be an application that allows the two HMDs to chat in linkage with each other.

[0017] 1 illustrates HMDs 101, 102, 103, and 104 respectively attached to the heads of four users 901, 902, 903, and 904, and a coordination mark 810. When there is no need to distinguish between them, they are represented by an HMD 100 and a user 900.

[0018] Information is transmitted and received between the HMDs 100 via wireless communications 912, 913, 914, 923, 924, and 934. Note that information may be transmitted and received between the HMDs 100 via a server or the like, rather than via direct wireless communications between the HMDs 100.

[0019] In this figure, the HMDs 101, 102, and 103 detect that the users 901, 902, and 903 have recognized a linkage mark 810 in their respective lines of sight 941, 942, and 943, and identify the app corresponding to the linkage mark 810 as an app to be executed in linkage (linked app).

[0020] The HMDs 101, 102, and 103 are HMDs 100 that have detected the same coordination mark 810, and are identified as a coordination group that coordinates and executes a coordinated app via wireless communications 912, 913, and 923. Then, the HMDs 101, 102, and 103 identified as the coordination group coordinate and execute a coordinated app between them.

[0021] 1, the linked application corresponding to the link mark 810 is a chat application. In this case, the HMDs 101, 102, and 103 execute the corresponding chat application in a linked manner. A chat screen showing the contents of the conversation between the users 900 is displayed on the display 131 of the HMDs 101, 102, and 103.

[0022] On the other hand, the user 904 of the HMD 104 whose line of sight 944 is not directed toward the coordination mark 810 does not recognize the coordination mark 810. That is, the HMD 104 does not detect the coordination mark 810. Therefore, the HMD 104 does not identify the app corresponding to the coordination mark 810 as a coordination app. Therefore, the HMD 104 does not join a coordination group that executes coordination apps in coordination. Therefore, a chat screen showing the contents of the conversation between the users 900 is not displayed on the display 131 of the HMD 104.

[0023] For example, in a chat application, it is usually necessary to select a chat partner or a chat group after starting the application. However, in this embodiment, when a user 900 who wants to participate simply sees a cooperation mark 810 associated with the chat application, the chat application is automatically started, and the HMDs 100 of the users 900 who are seeing the same cooperation mark 810 are recognized as a cooperation group. In this way, according to this embodiment, a cooperation group that cooperates and executes a cooperation application is automatically formed only by the HMDs 100 worn by the users 900 who are seeing the same cooperation mark 810 at that time. In addition, the cooperation application is specified simply by the user 900 wearing the HMD 100 recognizing (detecting) the cooperation mark 810. Therefore, it is not necessary to select members of the cooperation group or start the cooperation application.

[0024] Furthermore, the HMD 100 of the user 900 who does not see the coordination mark 810 does not join the members of the coordination group. Therefore, the HMD 100 of an unintended user 900 is not added to the members of the coordination group by mistake. That is, only the HMD 100 of the user 900 present at the site participates in the coordination group. Therefore, there is no risk of accidentally sending information to the HMD 100 of the user 900 who is not present at the site, and the coordination group can be safely formed. Furthermore, even in a situation where the user 900 is frequently replaced, the constituent members of the coordination group can be easily managed.

[0025] [System Configuration] The system configuration of the information processing system 700 of this embodiment that realizes the above will be described below. Fig. 2 is a system configuration diagram of the information processing system 700 of this embodiment.

[0026] As shown in this figure, an information processing system 700 of this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a coordination mark 810.

[0027] The HMD 100 is a device that is worn on the head of a user 900 and displays processed information on a display 131. The HMD 100 of this embodiment has a eyeglasses shape and includes a display 131 at the lens position of the eyeglasses. The display 131 may be a transmissive type or a non-transmissive type.

[0028] In the HMD 100, the user 900 can observe the situation in the real space via the display 131. In addition, an augmented reality (AR) object can be displayed on the display 131. Therefore, the user 900 can simultaneously view both the AR object displayed on the display 131 and the situation in the real space.

[0029] As described above, the linkage mark 810 is a mark that is detected and serves as a reference for identifying a linked application and certifying members of a linkage group. As described above, for example, an icon figure of a linked application is used. In this embodiment, the linkage mark 810 is fixed to a predetermined position, such as a wall, a pillar, or a desk.

[0030] [HMD] Next, the hardware configuration and functional blocks of the HMD 100 of this embodiment will be described with reference to the drawings.

[0031] [Hardware configuration] The hardware configuration of the HMD 100 of this embodiment that realizes the above processing will be described below. Fig. 3 is a block diagram showing an example of the configuration of the HMD 100 of this embodiment. In these figures, the same reference numerals are used to designate the same components as those shown in Figs. 1 and 2.

[0032] As shown in the figure, the HMD 100 of this embodiment includes a main processor 111, a system bus 112, a storage device 110, an operation reception device 120, an image processing device 130, a sound processing device 140, a sensor 150, a communication device 160, an expansion interface (I / F) unit 171, a timer 172, and a vibration generating device 173. Each component is connected to each other via the system bus 112, except for the transmitting and receiving antennas.

[0033] The main processor 111 is a main control unit that controls the entire HMD 100 according to a predetermined program. The main processor 111 is realized by a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 111 performs operation control processing of the entire HMD 100 by executing programs such as an operating system (OS) and various operation control applications stored in the storage device 110. The main processor 111 also controls the startup operation of various applications. The main processor 111 performs processing according to a clock signal measured and output by a timer 172.

[0034] The system bus 112 is a data communication path for transmitting and receiving data between the main processor 111 and each unit in the HMD 100 .

[0035] The storage device 110 includes a RAM 113, a ROM 114, and a flash memory 115. The storage device 110 stores programs such as an operating system and various operation control applications for music, images, documents, etc. In addition, base data required for basic operations by the operating system and other applications, and information data such as file data started by various applications and other applications are stored. For example, when a music application installed in the HMD 100 is started and music file data is selected, the selected music file data is played on the HMD 100, and the desired music can be listened to.

[0036] The RAM 113 is a program area for executing basic operation programs and other application programs. The RAM 113 is also a temporary storage area for temporarily storing data as necessary when various application programs are executed. The RAM 113 may be integrated with the main processor 111.

[0037] The ROM 114 and the flash memory 115 store various programs for implementing the functions of the HMD 100, operation setting values, sensor information including detection values ​​from a sensor 150 described later, and various display data such as virtual objects and content.

[0038] Flash memory 115 stores operation programs downloaded from network 300 and various data created by the operation programs. Each operation program stored in flash memory 115 can be updated and its functions expanded by downloading from each server device on network 300.

[0039] Furthermore, flash memory 115 can store content such as moving images, still images, and audio downloaded from network 300. It also stores data such as moving images and still images captured by in-camera 134 or out-camera 133.

[0040] The ROM 114 and the flash memory 115 are so-called non-volatile storages that retain stored information even when power is not being supplied from an external source to the HMD 100. Hereinafter, unless there is a need to distinguish between the ROM 114 and the flash memory 115, they will be collectively referred to as the internal memory storage. The main processor 111 loads new application programs stored in the internal memory storage into the RAM 113 and executes them, thereby enabling the HMD 100 to realize various functions.

[0041] The built-in memory storage needs to retain stored information even when power is not being supplied to the HMD 100. Therefore, for the built-in memory storage, devices such as flash ROM, SSD (Solid State Drive), and HDD (Hard Disc Drive) are used.

[0042] The built-in memory storage stores programs such as the operating system and various operation control applications such as chat, music, images, documents, etc. It also stores information data such as base data required for basic operations by the operating system and file data launched by various applications.

[0043] For example, when a chat application installed in the HMD 100 is started and various file data necessary for chat operations, such as chat partners and previous chat contents, is selected, the user can talk with the desired chat partner using the selected chat file data in the HMD 100. Also, when a music application installed in the HMD 100 is started and music file data is selected, the selected music file data is played in the HMD 100, allowing the user to listen to the desired music.

[0044] The operation reception device 120 receives input of an operation instruction for the HMD 100. In this embodiment, the operation reception device 120 includes operation keys 121. The operation keys 121 include a power key, a volume key, a home key, and the like.

[0045] It should be noted that the HMD 100 of this embodiment does not necessarily have to include all of these operation reception devices 120. The operation reception device 120 may be provided in a position and shape within the HMD 100 that allows the user 901 to easily perform input operations.

[0046] The operation reception device 120 may also receive the operation of the HMD 100 via a separate information processing terminal device connected by wired or wireless communication. In this case, for example, an input means such as a keyboard, key buttons, or touch keys may be provided. The operation reception device 120 may also use the line of sight of the user 901. For example, the operation reception device 120 may display an input operation screen in the display 131 and capture the input operation information based on the position on the input operation screen to which the line of sight of the user 901 is directed, detected by the in-camera 134 described later. A pointer may be displayed on the input operation screen and the input operation information may be captured by operating the pointer. The user 901 may also utter a voice indicating an input operation, and the input operation information may be captured by collecting the voice with the microphone 143 described later. The out-camera 133 and various sensors 150 may also be used to detect the hand or body movements (gestures) of the user 901 and capture them as input operation information.

[0047] The image processing device 130 is an image (video) processor, and includes a display 131, an outer camera 133, and an inner camera 134.

[0048] The display 131 is a display device (display) such as a liquid crystal panel, and presents image data processed by the image signal processing unit to a user of the HMD 100. The display 131 includes a left-eye display and a right-eye display. The display 131 may be a transmissive (optically transmissive) display or a non-transmissive (video transmissive) display.

[0049] The optically transmissive display includes a projection unit that projects various information such as information reproduced by the activated application and notification information for the user 901, and a transparent half mirror that forms an image of the various projected information in front of the user's eyes. The video transmissive display also includes a liquid crystal panel or the like that displays various information together with real-space objects in front of the user's eyes photographed by the out-camera 133. Through the display 131, the user 901 visually recognizes information reproduced by the activated application, such as chat conversations, music, images, and documents, in addition to images in front of the user's eyes within the field of view.

[0050] The display 131 displays notification information to the user, such as the remaining battery capacity, various alarms, and the time, and also displays within the display 131 the icon of an application to be started.

[0051] The image signal processing unit is an image (video) signal processor that processes images input from out-camera 133 and in-camera 134. The image signal processing unit also superimposes an object created by main processor 111 or the like on the input image and outputs the superimposed image to display 131. The image signal processing unit may be realized by main processor 111, or may be realized by a processor dedicated to images provided separately from main processor 111.

[0052] Outer camera 133 and in-camera 134 are cameras that input image data of the surroundings and objects by converting light input from a lens into an electrical signal using electronic devices such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0053] The outer camera 133 captures an image of the surroundings of the HMD 100. The outer camera 133 is installed in front of the HMD 100 and captures the forward visual field. The HMD 100 is classified into an optical transmission type and a video transmission type depending on the type of the display 131, but both types capture and import the visual field in front of the eyes captured by the outer camera 133.

[0054] The in-camera 134 captures an image of a region different from that of the out-camera 133. For example, the in-camera 134 captures an image of the eyes of the user 901. In this embodiment, the in-camera 134 functions as a gaze detection sensor together with a processing unit that performs gaze identification processing.

[0055] The gaze detection sensor includes, for example, a right eye gaze detection unit and a left eye gaze detection unit, which detect the gaze of the right eye and the left eye, respectively. The process of detecting the gaze may utilize well-known technology that is generally used as eye tracking processing. For example, in a method using corneal reflex, an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of ​​the reflected light (corneal reflex) created by the infrared LED irradiation is used as a reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex. In this case, an infrared camera and an infrared LED are provided.

[0056] The operation reception device 120 may display an input operation screen within the display screen of the display 131 and capture input operation information based on the position on the input operation screen to which the gaze detected by the right eye gaze detection unit and the left eye gaze detection unit is directed. A pointer may be displayed on the input operation screen and the operation reception device 120 may operate the pointer to capture the input operation information.

[0057] The audio processing device 140 is an audio processor that processes audio, and includes a speaker 141 and a microphone 143.

[0058] The speaker 141 outputs the audio signal processed by the audio signal processing unit to the outside. The speaker 141 outputs audio to the outside to inform the user of audio such as notification information and music. The audio signal processing unit is an audio signal processor. In the HMD 100, for example, headphones are used.

[0059] The microphone 143 collects the user's voice and the like, converts it into audio data, and inputs it. For example, an external sound microphone and a speech sound microphone may be provided. These microphones collect sounds from the outside and the voice of the user 901 himself, respectively. Headphones may be connected. The headphones are worn on the ears of the user 901 to listen to the voice to be spoken to the user 901. The headphones can notify the user 900 of notification information by voice. As the operation reception device 120, the user 900 may speak a voice indicating an input operation, and the voice may be collected by the microphone 143 to capture the input operation information.

[0060] The sensor 150 is a group of sensors for detecting the state of the HMD 100. In this embodiment, the sensor 150 includes a GPS (Global Positioning System) receiver 151, a gyro sensor 152, a geomagnetic sensor 153, an acceleration sensor 154, and a depth sensor 155. These sensors detect the position, movement, inclination, direction, and the like of the HMD 100. The depth sensor 155 also acquires distance information from the HMD 100 to an object. Note that other sensors may also be included.

[0061] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, impact, etc. The gyro sensor 152 is a sensor that detects angular velocity in the rotation direction, and can detect vertical, horizontal, and diagonal posture states. The movement of the HMD 100 can be detected using the acceleration sensor 154 and gyro sensor 152 mounted on the HMD 100.

[0062] The geomagnetic sensor 153 is a sensor that detects the magnetic force of the earth and detects the direction in which the HMD 100 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions may be used to detect the movement of the HMD 100 by capturing changes in the geomagnetism in response to the movement of the HMD 100.

[0063] The depth sensor 155 is a sensor that can capture the shape of a target object such as a person or an object in three dimensions. A LiDAR (Light Detection and Ranging) sensor that irradiates an object with infrared or other laser light, measures the scattered light that bounces back, and analyzes and detects the distance to a distant target object and the state of the target object, a TOF (Time Of Flight) sensor that measures the distance by measuring the reflection time of pulsed light irradiated on the subject for each pixel, and a millimeter wave radar that emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the target object and the state of the target object, etc., are used.

[0064] The communication device 160 is a communication interface that performs wireless communication with other nearby information terminals by short-distance wireless communication, wireless LAN, or base station communication. During wireless communication, data is transmitted and received via a transmitting / receiving antenna. In this embodiment, participation ID information that collects all IDs of each information terminal participating in the cooperative execution of the application, application start instruction information, file data to be played by the started application, and the like are transmitted and received.

[0065] The communication device 160 includes a LAN (Local Area Network) communication interface (I / F) 161, a telephone network communication I / F 162, and a short-range wireless communication I / F 163.

[0066] The LAN communication I / F 161 is connected to a network 300 such as the Internet via a wireless access point or the like, and transmits and receives data to and from each server on the network 300. The connection to the wireless access point or the like may be made using Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0067] The telephone network communication I / F 162 is connected to a communication network using a mobile telephone communication network, and transmits and receives information to and from a server on the communication network. The communication methods used include, for example, third generation mobile communication systems (hereinafter referred to as "3G") such as the Global System for Mobile Communications (GSM) (registered trademark), Wideband Code Division Multiple Access (W-CDMA), CDMA2000, and Universal Mobile Telecommunications System (UMTS), as well as communication methods called the Long Term Evolution (LTE), fourth generation (4G), and fifth generation (5G).

[0068] The short-distance wireless communication I / F 163 is an interface for short-distance wireless communication. Short-distance wireless communication is a communication method for directly transmitting and receiving data between surrounding HMDs 100. In this embodiment, the communication is performed using, for example, an electronic tag. The electronic tag is a communication method that uses RFID (Radio Frequency Identification) technology that uses radio waves to read and write data in an IC chip in a non-contact manner.

[0069] However, the short-distance wireless communication is not limited to this, and may be a method that allows wireless communication at least when the HMD 100 is near another information terminal. For example, communication according to the Bluetooth standard, infrared communication using IrDA (Infrared Data Association, registered trademark) or the like, or communication using Zigbee (registered trademark) or HomeRF (Home Radio Frequency, registered trademark) may be used. Alternatively, communication may be performed using a wireless LAN such as Wi-Fi (registered trademark). Furthermore, an ultra-wideband wireless system (Ultra Wide Band: UWB) may be used.

[0070] The LAN communication I / F 161, the telephone network communication I / F 162, and the short-range wireless communication I / F 163 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0071] The communication device 160 may use other methods such as optical communication or sound wave communication as a means of wireless communication. In that case, a light emitting / receiving unit and a sound wave output / sound wave input unit are used instead of a transmitting / receiving antenna.

[0072] In addition, in the case of image applications that handle high-definition images, the amount of data used is dramatically large, and in this case, high-speed, large-capacity communication networks such as 5G (5th Generation: 5th generation mobile communication system) and local 5G may be used for wireless communication. This makes it possible to prevent display delays due to increased data volume and quality degradation due to data loss during transmission, improving usability.

[0073] The expansion interface unit 171 is a group of interfaces for expanding the functions of the HMD 100, and in this embodiment includes a charging terminal, a video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The video / audio interface inputs video signals / audio signals from external video / audio output devices, outputs video signals / audio signals to external video / audio input devices, etc. The USB interface connects SB devices. The memory interface connects memory cards and other memory media to send and receive data.

[0074] The vibration generator 173 generates vibrations under the control of the main processor 111, and converts notification information sent to the user by the HMD into vibrations. The vibration generator 173 can reliably convey notifications to the user by generating vibrations on the user's head, which is closely fitted to the user's head.

[0075] 3 includes components that are not essential to this embodiment, but the effects of this embodiment are not impaired even if these components are not provided. In addition, components not shown, such as a digital broadcast receiving function and an electronic money settlement function, may be added.

[0076] [HMD functional blocks] Next, the functional configuration of the HMD 100 of this embodiment will be described. Fig. 4(a) is a functional block diagram of the configuration of the HMD 100 according to this embodiment. As shown in this figure, the HMD 100 includes a main control unit 210, a detection unit 220, an execution unit 230, and a collaboration unit 240. In addition, the HMD 100 includes an application list 250, a collaboration group list 260, and application / processing data 270 as data used in processing.

[0077] The main control unit 210 controls the overall operation of the HMD 100. By executing programs such as an operating system (OS) and various operation control applications, the main control unit 210 controls the overall operation of the HMD and also controls the launch and execution of various applications.

[0078] The detection unit 220 detects the linkage mark 810 associated with the application. In this embodiment, as shown in Fig. 5(a) and Fig. 5(b), the detection unit 220 captures the surroundings (field of view) of the HMD 100 with the out-camera 133 and analyzes the obtained image (captured image 136). Then, when the captured image includes an area 137 that matches a figure registered in association with the application in the application list 250 at a predetermined ratio, it is determined that the linkage mark 810 has been detected.

[0079] Here, the application list 250 will be described. The application list 250 is a list in which data (mark data) of the linkage mark 810 is stored in association with an application. In this embodiment, as shown in Fig. 4(b), mark data 252 is registered in association with application identification information (application ID (identification) 251) that is information for identifying an application. The mark data 252 includes, for example, mark shape data 253 and mark identification information (mark ID 254) that is information for identifying the mark.

[0080] The application list 250 is registered, for example, when an application is installed. The mark data 252 may be, for example, icon data used to start an application. The mark data 252 may be updated (changed or added) when an application is updated. The mark data 252 is not limited to one type for one application.

[0081] When the HMD 100 is not executing an association application, the detection unit 220 analyzes the captured image 136 at a predetermined time interval and determines whether or not the captured image 136 includes an area 137 corresponding to the shape data 253 registered in the application list 250. The determination of whether or not the captured image 136 includes an area 137 (figure) corresponding to the association mark 810 is performed by pattern matching or the like. If included, it is determined that the association mark 810 has been detected. Note that the detection operation by the detection unit 220 may be started by an instruction from the user 900 for purposes such as reducing power consumption, not wanting to perform association even if the association mark 810 is recognized, etc.

[0082] When the linkage mark 810 is detected, the detection unit 220 specifies an application registered in the application list 250 in association with the detected mark data 272 as an application to be executed in a linked manner (linkage application).

[0083] The detection unit 220 may perform the above analysis when it receives a mark detection instruction from the user 900.

[0084] Furthermore, for example, when a mark that is not stored as mark data 272 in the application list 250 is extracted from the captured image 136, the detection unit 220 may inquire of a server on the network and determine whether or not there is an application associated with the mark. If there is an application associated with the mark, the application is downloaded together with the mark, and the mark is stored in the application list 250 in association with the application ID 251. The application itself is stored in the application / processing data 270.

[0085] In addition, when detecting, in order to determine whether the linkage mark 810 corresponds to a linkage app or is simply a figure, criteria may be set in advance for the linkage mark 810, such as size, shape, frame type, etc.

[0086] Alternatively, instead of using the entire field of view of the outer camera 133 as the detection range, as shown in Fig. 5(c), it may be determined whether or not the detection has been successful based on whether or not there is a corresponding link mark 810 in a predetermined area (linked field of view 131a) in the field of view of the outer camera 133. The linked field of view 131a may be determined in advance, or may be a predetermined range centered on the intersection of the line of sight of the user 900 and the field of view of the outer camera 133, for example.

[0087] The execution unit 230 starts and executes the application identified as the cooperative application by the detection unit 220. The application is stored in the application / processing data 270 in association with the application ID 251 identified by the detection unit 220. When the application is executed, it is executed in cooperation with another HMD 100.

[0088] The application / processing data 270 is the application and various file data required for executing the application. Using this data, the execution unit 230 can launch and execute a desired linked application. During the execution in cooperation, the execution unit 230 transmits and receives information between the multiple HMDs 100 via the communication device 160.

[0089] The collaboration unit 240 manages, as members of a collaboration group, the HMDs 100 that have detected the same collaboration mark 810. In this embodiment, the collaboration unit 240 manages the relevant HMDs 100 among the HMDs 100 that are within the communication range of short-distance wireless communication and can communicate with each other.

[0090] When a change occurs in the state (status) of the HMD 100, the collaboration unit 240 generates notification data notifying the change and transmits it to the surrounding HMDs 100. In this embodiment, for example, when the collaboration unit 240 detects the collaboration mark 810, notification data indicating the detection is generated. Also, when the collaboration unit 240 ends the running linked app, notification data indicating the end is generated.

[0091] Here, the notification data 280 generated by the linking unit 240 will be described. As shown in Fig. 4(c), the notification data 280 registers a mark ID 281 which is information for identifying the detected linking mark 810, terminal identification information (device ID 282) which is information for identifying the HMD 100 itself, an address 283 of the HMD 100, an application ID 284 which is information for identifying the linked application, and information indicating a state (here, detection or end; status 285). The device ID 262 of each device of the HMD 100 is set in advance. The address 283 is used, for example, when linking applications is performed via a network such as a LAN. The address 283 may be used as the device ID 282.

[0092] For example, when the detection unit 220 detects the linkage mark 810, notification data 280 in which "detected" is registered in the status 285 (hereinafter, detection notification data 280a) is generated and transmitted. On the other hand, when the linkage app is terminated, notification data 280 in which "ended" is registered in the status 285 (hereinafter, end notification data 280b) is generated and transmitted. The notification data 280 is transmitted, for example, by short-distance wireless communication.

[0093] In this embodiment, the detection notification data 280a also serves as inquiry data inquiring whether or not the same coordination mark 810 has been detected. When the coordination unit 240 receives detection notification data 280a from another HMD 100 in which the mark ID 281 is the same as the mark ID 254 of the coordination mark 810 detected by the own device, the coordination unit 240 returns the detection notification data 280a to the sending HMD 100 to indicate that the own device has also detected the same coordination mark 810. In this way, when joining a coordination group midway, information on all HMDs 100 already participating in the group can be obtained.

[0094] Furthermore, when the collaboration unit 240 detects the collaboration mark 810, it generates and manages a collaboration group list 260. The collaboration group list 260 is a list that manages the HMDs 100 that execute the collaboration app in collaboration as members of a collaboration group, as shown in Fig. 4(d). The collaboration group list 260 includes a mark ID storage location 261 that stores the mark ID of the collaboration mark, and a device ID 262 and an address 263 of the HMDs 100 in the collaboration group.

[0095] When the collaboration mark 810 is detected, the collaboration unit 240 first generates the collaboration group list 260 and registers the mark ID (A) corresponding to the detected collaboration mark 810 in the mark ID storage location 261 of the collaboration group list 260. After that, when the collaboration unit 240 receives the detection notification data 280a of the same mark ID (A) as the mark ID (A) registered in the mark ID storage location 261 of the collaboration group list 260, the collaboration unit 240 registers the device ID 282 and the address 283 included in the detection notification data 280a in the device ID 262 and the address 263 of the collaboration group list. In addition, when the collaboration unit 240 receives the end notification data 280b of the same mark ID (A) as the mark ID (A) registered in the mark ID storage location 261 of the collaboration group list 260, the collaboration unit 240 deletes the data having the device ID 282 and the address 283 included in the end notification data 280b from the collaboration group list 260.

[0096] The collaboration unit 240 manages the HMDs 100 belonging to the collaboration group by transmitting and receiving the notification data 280 and updating the collaboration group list 260 accordingly. In this embodiment, the collaboration group list 260 is generated and managed in the same manner in all HMDs 100 that detect the same collaboration mark 810 and execute the collaboration application. As a result, the collaboration group list 260 is shared among the HMDs 100.

[0097] The above functions are realized by the main processor 111 loading a program stored in the ROM 114 into the RAM 113 and executing it. The lists and data are stored in the storage device 110.

[0098] With the above configuration, in the HMD 100, the outer camera 133 captures the field of view in front of the HMD 100, and when a linking mark 810 such as an icon is captured within the captured field of view or within the linking field of view 131a, an application associated with the linking mark 810 can be executed in linkage as a linking app.

[0099] [Linkage processing] Next, the flow of the cooperative processing of this embodiment will be described with reference to Fig. 6. Here, the processing in one HMD 100 will be described.

[0100] When the detection unit 220 detects the linkage mark 810 (step S1101), the detection unit 220 identifies an application registered in the application list 250 in association with the detected linkage mark 810 as a linkage application (step S1102).

[0101] The collaboration unit 240 generates the collaboration group list 260 (step S1103), and transmits the detection notification data 280a to the surrounding HMDs 100 by short-distance wireless communication (step S1104). Note that the creation of the collaboration group list 260 and the transmission of the detection notification data 280a may be performed in any order.

[0102] The execution unit 230 launches the linked application, and executes the linked application until an instruction to end the linked application is received (steps S1105 and S1106).

[0103] During execution of the linked application, when detection notification data 280a is received from the surrounding HMD 100 (step S1107), if the detection notification data 280a has the same mark ID 281 as the linked mark 810 detected by the link unit 240 itself, the link unit 240 updates the linked group list 260 (step S1108). Here, the device ID 282 and address 283 of the detection notification data 280a are added to the linked group list 260. Then, after that, the linked application is executed based on the updated linked group list 260.

[0104] Furthermore, when an instruction to end the linked application is received, the execution unit 230 ends the linked application, and the link unit 240 transmits end notification data to the surrounding HMDs 100 (step S1109), and ends the process.

[0105] As described above, the information processing system 700 of this embodiment executes an application in cooperation with the HMDs 100, which are multiple information processing terminals. Each HMD 100 includes a detection unit 220 that detects a predetermined coordination mark 810, and an execution unit 230 that executes an application associated with the detected coordination mark 810 as a coordination app in cooperation with other HMDs 100 that have detected the same coordination mark 810. Furthermore, each HMD 100 of this embodiment includes a coordination unit 240 that manages other HMDs 100 that have detected the same coordination mark 810 as constituent terminals (constituent members) of a coordination group that executes a coordination app in cooperation with each other.

[0106] In this manner, according to the present embodiment, by the user 900 recognizing the same linkage mark 810, it is possible to automatically identify the linked app to be started and executed on a plurality of HMDs 100 and to identify the linked HMDs 100. That is, according to the present embodiment, simply by detecting the same linkage mark 810 on each HMD 100, a linkage group can be formed and the same or similar linked app can be executed in a linked manner. For example, a group can be formed among users 900 wearing information terminals such as the HMD 100, and chats, meetings, games, and the like can be easily and conveniently performed among the group.

[0107] Moreover, the HMD 100 that does not detect the coordination mark 810 does not join the coordination group. Therefore, coordination with the HMD 100 of an unintended user 900 is not performed, and safety is ensured.

[0108] Conventionally, when an HMD 100 that was not a member of a cooperation group at the time of initial setup joins later, the cooperation group needs to be reconstructed. When reconstructing the group, the members of the cooperation group need to take the trouble of adding the person who will join later to the group. However, according to this embodiment, the latest members can be managed in a cooperation group list, so that the person who will join later can join the cooperation group simply by looking at the cooperation mark 810. Therefore, the operations that were conventionally required are no longer necessary, and usability can be significantly improved.

[0109] In this manner, according to this embodiment, application linkage can be realized between a plurality of information processing terminals in a simple and user-friendly manner.

[0110] In addition, when launching a linking app using the method of this embodiment, the HMD 100 to be linked may be limited to one that communicates only within a local network with a limited connection range, such as a local area network (LAN), a personal area network (PAN), or a virtual private network (VPN), for security reasons.

[0111] <Variation 1> In the above embodiment, each HMD 100 starts the linked app after identifying the linked app, but this is not limited to the above. For example, the HMD 100 that first detects the link mark 810 may be set as a main device, and the main device HMD 100 may transmit a link app start instruction to the other HMDs 100, and the other HMDs 100 may start the linked app after receiving the start instruction.

[0112] In this case, for example, the HMD 100 that detects the coordination mark 810 earliest is set as the main device. In this case, when the coordination unit 240 identifies a coordination app and has not yet received the detection notification data 280a, it transmits the detection notification data to the surrounding HMDs 100 and sets a flag that designates the HMD 100 itself as the main device. Then, the coordination unit 240 of the main device generates a coordination group list 260, and registers the HMD 100 in the coordination group list 260 every time it receives the detection notification data 280a from the surrounding HMDs 100 for a predetermined period of time. After the predetermined period has elapsed, the coordination unit 240 outputs an application start instruction to the HMD 100 registered in the coordination group list 260.

[0113] After detecting the coordination mark 810, the coordination unit 240 of the HMD 100 other than the main device receives, from the surrounding HMD 100, detection notification data 280a of the same mark ID (A) as the mark ID (A) corresponding to the coordination mark 810, and generates detection notification data 280a of the own device as a reply and transmits it to the HMD 100 (main device) that sent it. After that, when an application start instruction is received, the execution unit 230 starts and executes the coordination application.

[0114] In addition, when a main device is provided, all HMDs 100 in the linked group may have the linked group list 260, or only the main device may have the linked group list 260. In the case where only the main device has the linked group list 260, when a linked application is executed, the linking process is performed via the main device.

[0115] Furthermore, the main device is not limited to the HMD 100 that detects the coordination mark 810 first. For example, the main device may be determined in advance.

[0116] In this modified example, when the main device transmits the detection notification data 280a, the detection notification data 280a is received as a reply from the HMD 100 that detects the same coordination mark 810, but the present invention is not limited to this. For example, the main device may be configured to inquire of all HMDs 100 in a predetermined search range as to whether or not they recognize the coordination mark 810. Then, the HMDs 100 that receive a response indicating that they have detected the coordination mark 810 are registered in the coordination group list 260.

[0117] In this case, the search time can be shortened by limiting the search range to a predetermined range. For example, the search range can be limited to between devices in the above-mentioned LAN, PAN, or VPN, between HMDs 100 that can directly communicate with each other via short-distance wireless communication (Wi-fi (registered trademark), Bluetooth, etc.), between HMDs 100 connected to the same access point, between HMDs 100 whose network hop count (the number of intermediate network devices that data must pass through between the source and destination) is within a specified range, etc. Also, the search may be configured to be terminated at a specified time. This will necessarily exclude HMDs 100 with a large number of hops from the search target, thereby achieving the same purpose.

[0118] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, collaboration is realized only by the HMD 100. Also, except for modified examples, basically, each HMD 100 manages a collaboration group. On the other hand, in this embodiment, a server, which is a second information processing terminal different from the HMD 100, manages the collaboration group and realizes collaboration. Hereinafter, this embodiment will be described with a focus on the configuration different from the first embodiment.

[0119] 7, an information processing system 701 of this embodiment further includes a server 400 in addition to the configuration of the information processing system 700 of the first embodiment. The multiple HMDs 100 can transmit and receive data to and from each other through short-range wireless communication. Also, each of the multiple HMDs 100 can transmit and receive information to and from the server 400 via a network 300 such as a LAN. The HMDs 100 are connected to the network 300 via, for example, a wireless router or the like.

[0120] The hardware configuration of the HMD 100 of this embodiment is the same as that of the first embodiment. The functional configuration of the HMD 100 of this embodiment is also basically the same as that of the first embodiment. However, the configurations of the link unit 240 and the execution unit 230 are different. The data stored in the storage device 110 is also different.

[0121] In this embodiment, similarly to the first embodiment, when the linkage mark 810 is detected, the linkage unit 240 transmits detection notification data 280a to the server 400. When the linkage app is terminated, the linkage unit 240 transmits termination notification data 280b to the server 400. Note that, in this embodiment, the detection notification data 280a may also include current position information of the HMD 100.

[0122] In this embodiment, the server 400 manages a group of HMDs 100 that cooperate to execute a cooperative application as a cooperative group.

[0123] The server 400 may be of any form as long as it can be connected to each HMD 100 via communication and send and receive necessary information, and may be, for example, a local server, a cloud server, an edge server, a network service, or the like.

[0124] 8(a) is a block diagram of functions related to the processing of the present embodiment of the server 400 of the present embodiment. As shown in this figure, the server 400 includes a main control unit 410, a transmission / reception unit 420, a cooperation control unit 430, and a cooperation management unit 440. In addition, the server 400 manages a cooperation group list 460.

[0125] The transmission / reception unit 420 receives the notification data 280 from each HMD 100 .

[0126] The cooperation management unit 440 executes cooperation management processing to manage a group of HMDs 100 that cooperate to execute the same cooperation app as a cooperation group. In this embodiment, one cooperation group list 460 is generated for a group of HMDs 100 that detect the same cooperation mark 810, and the cooperation group is managed by the list.

[0127] The cooperation group list 460 is generated for each mark ID. Each cooperation group list 460 basically has the same configuration as the cooperation group list 260 of the first embodiment. That is, as shown in Fig. 8(b), for each mark ID that specifies the cooperation mark 810, the cooperation group list 460 includes a mark ID storage location 461 that stores the mark ID, and a device ID 462 and an address 463 of the HMD 100 that executes the cooperation app in cooperation. In this embodiment, when the cooperation group list 460 is received from the server 400, the cooperation group list 260 stored in the storage device 110 is replaced with the latest one.

[0128] When the execution of the linked app is performed via the server 400, the linkage control unit 430 controls the execution of the linked app among the multiple HMDs 100. The execution of the linked app is controlled based on the linkage group list 460. For example, when data is received, the linkage control unit 430 transmits the data to HMDs 100 other than the source HMD 100 in the linkage group including the device ID of the source HMD 100. The linkage control unit 430 is used in a modified example described later.

[0129] 8(c) is a hardware configuration diagram of the server 400. The server 400 of this embodiment includes a CPU 401, a memory 402, a storage device 403, and a communication interface 404. Each of the above functions is realized by the CPU 401 loading a program stored in the storage device 403 into the memory 402 and executing the program.

[0130] With the above configuration, in this embodiment as well, a collaboration group can be formed between a plurality of HMDs 100, and a common application can be executed in collaboration with each other.

[0131] [Linkage processing] Next, a flow of processing when executing an integrated application in this embodiment (integrated application execution processing) will be described with reference to Fig. 9. Here, data transmission and reception with one HMD 100 will be described as an example. Data transmission and reception with other HMDs 100 is similar.

[0132] When the detection unit 220 of the HMD 100 detects the coordination mark 810 (step S1101), the detection unit 220 identifies an application registered in association with the detected coordination mark 810 as a coordination application (step S1102).

[0133] The collaboration unit 240 of the HMD 100 generates the detection notification data 280a and transmits it to the server 400 (step S2104).

[0134] When the collaboration management unit 440 of the server 400 receives the detection notification data 280a (step S2201), the collaboration management unit 440 updates the collaboration group list 460 by storing the data of the HMD 100 that is the transmission source in the collaboration group list 460 (step S2202). Here, the device ID 282 and the address 283 are added to the collaboration group list 460 in which the mark ID corresponding to the mark ID 281 of the detection notification data 280a is stored in the mark ID storage location 461. If there is no collaboration group list 460 in which the mark ID corresponding to the mark ID 281 of the detection notification data 280a is stored in the mark ID storage location 461, a new one is created and then added.

[0135] Then, when the cooperation management unit 440 finishes updating, it returns data (start-up instruction) indicating that the process of joining the cooperation group is completed and instructing startup to the transmission source HMD 100 (step S2203). In addition, it transmits the cooperation group list to all HMDs 100 registered in the cooperation group list.

[0136] When the execution unit 230 of the HMD 100 receives the start instruction, the execution unit 230 of the HMD 100 starts and executes the linked application (step S1105). As in the first embodiment, the execution unit 230 of the HMD 100 continues to execute the executed application until it receives an instruction to end the executed application (step S1106).

[0137] During execution of the linked application, in this embodiment, the execution unit 230 transmits and receives data required for linked execution to and from other HMDs 100 registered in the linked group list (step S2108).

[0138] When the execution unit 230 of the HMD 100 receives an instruction to end the linked application from the user 900, the execution unit 230 ends the linked application, and the collaboration unit 240 generates end notification data 280b, transmits it to the server 400 (step S2109), and ends the process. On the other hand, the collaboration management unit 440 of the server 400 that received the end notification data 280b (step S2205) updates the linked group list 460 by deleting the HMD 100 that sent the data from the linked group list 460 (step S2206). Furthermore, the updated linked group list 460 is transmitted to each HMD 100 registered in the updated linked group list 460.

[0139] The same is true for other HMDs 100, and when the collaboration mark 810 is detected, the detection notification data 280a is transmitted to the server 400. When the server 400 receives the detection notification data 280a or the end notification data 280b, the server 400 updates the collaboration group list 460 as needed, and returns the updated collaboration group list 460.

[0140] As described above, the information processing system 701 of the present embodiment executes an application in cooperation with the HMDs 100, which are multiple information processing terminals. Each HMD 100 includes a detection unit 220 that detects a predetermined cooperation mark 810, and an execution unit 230 that executes an application associated with the detected cooperation mark 810 as a cooperation app in cooperation with another HMD 100 that has detected the same cooperation mark 810. Furthermore, the information processing system 701 of the present embodiment further includes a server 400 that includes a cooperation management unit 440 that manages the multiple HMDs 100 that have detected the same cooperation mark 810 as members of a cooperation group that cooperate to execute a cooperation app. The cooperation management unit 440 transmits data obtained by the cooperation management process to the other HMDs 100, and receives data obtained by the cooperation management process from the other HMDs 100. Meanwhile, the execution unit 230 of the HMD 100 transmits data obtained by the processing of the cooperation app to the other HMDs 100, and receives data obtained by the processing of the cooperation app from the other HMDs 100.

[0141] According to this embodiment, similar to the first embodiment, application linkage can be realized simply, safely, and conveniently. Furthermore, according to this embodiment, the linked HMDs 100 are managed by the server 400. As long as the HMDs 100 can detect the same linkage mark 810 and can communicate with the server 400, there are no restrictions on the placement of the HMDs 100, and a system with a higher degree of freedom can be realized.

[0142] <Variation 2> In the HMD 100 of the present embodiment, when a start instruction is received from the server 400, the linked app is started. However, this is not limited to this. For example, similar to the first embodiment, when a linked app is identified, the HMD 100 may start the linked app on its own. In this case, the server 400 does not need to transmit the start instruction.

[0143] <Modification 3> The start-up instruction may be transmitted simultaneously. For example, when a plurality of HMDs 100 visually recognize the cooperation mark 810 and form a cooperation group to execute a cooperation application at almost the same time, the server 400 transmits participation permission data to the plurality of HMDs 100 that are the transmission sources of the detection notification data 280a having the same mark ID 281 within a predetermined period of time.

[0144] <Modification 4> The server 400 manages each HMD 100 that detects the coordination mark 810 as a coordination group. However, the management of the coordination group is not limited to this.

[0145] For example, even if a group of HMDs 100 detects the same coordination mark 810, they may be divided into multiple coordination groups and managed. In this case, for example, they may be divided by the time when the detection notification data 280a is transmitted. Also, they may be divided by the position information of the HMDs 100.

[0146] In the case of dividing by time, for example, as described above, when the cooperation management unit 440 receives detection notification data 280a including the same mark ID 281 consecutively within a predetermined period, the cooperation management unit 440 makes the HMDs 100 that are the transmission sources of the detection notification data 280a into one cooperation group. After that, after the predetermined period has elapsed, the group of HMDs 100 that have transmitted detection notification data 280a including the same mark ID 281 are made into another cooperation group.

[0147] For example, as shown in Fig. 10, in the case where a group of a plurality of HMDs 100 sees the same cooperation mark 810 and starts chatting at different times, the cooperation management unit 440 divides the HMDs 100 into groups based on the timing at which the chat starts. Then, a cooperation group list 460 for each group is generated and managed. Here, a cooperation group list 460 for cooperation group 1 and a cooperation group list 460 for cooperation group 2 are generated and managed. Note that in this case as well, an HMD 104 in which the user 900 does not recognize the cooperation mark 810, i.e., an HMD 104 that does not detect the cooperation mark 810, does not join any cooperation group.

[0148] When dividing the group according to the position information of the HMD 100, the cooperation management unit 440 performs group division using the position information transmitted from the HMD 100. The boundaries for dividing the group are determined in advance.

[0149] <Variation 5> In the above embodiment, the linked app is executed between the HMDs 100 in the linked group, but this is not limiting. For example, the linked app may be configured to be executed via the server 400. In this case, the above-mentioned link control unit 430 controls the execution of the linked app. In addition, the transmission / reception unit 420 mediates the transmission and reception of data between the HMDs 100 when a plurality of HMDs 100 execute a linked app in cooperation with each other.

[0150] In this case, the cooperation group list 460 may be managed only in the server 400. That is, the cooperation management unit 440 does not need to transmit the cooperation group list 460 to the HMD 100 in step S2203 or step S2206.

[0151] In each HMD 100, while a linked application is being executed, the execution unit 230 transmits data required for the linked execution to the server 400. In addition, in the server 400, the linkage control unit 430 refers to the linkage group list 460, and when data is received from an HMD 100 in the linkage group, the linkage control unit 430 processes the data and transmits it to the other HMDs 100 in the linkage group.

[0152] The server 400 may store applications, files, and data on the server 400 and transfer them by communication to each HMD 100 as appropriate. An application may be started on the server 400, and playback information such as video, screen display, and audio may be transferred to the HMD 100 that is a member of the cooperative group.

[0153] <<Third embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the link mark to be recognized is displayed on the display of an information processing terminal such as a smart phone. In the following, this embodiment will be described assuming that the information processing terminal is a smart phone.

[0154] 11 is a diagram for explaining an overview of an information processing system 702 according to this embodiment. As shown in this figure, the information processing system 702 according to this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a smartphone 500.

[0155] In this embodiment, each HMD 100 determines whether or not it is in a linked state with the smartphone 500. Then, the smartphone 500 manages the group of HMDs 100 in a linked state with itself as members constituting a linked group.

[0156] An information processing system 702 of this embodiment includes a plurality of HMDs 100 (101, 102, 103, 104) and a smartphone 500. In this embodiment, the cooperation mark 820 is displayed on the display 531 of the smartphone 500.

[0157] The HMD 100 of this embodiment is basically the same as that of the first embodiment, except for the processing of the detection unit 220 and the linkage unit 240 and the data that is held.

[0158] The detection unit 220 of the HMD 100 in this embodiment first determines whether the smartphone 500 and the HMD 100 are in a predetermined positional relationship. That is, it determines whether the smartphone 500 is in a predetermined arrangement state in which it can operate in a cooperative operation mode described later. In this embodiment, as shown in FIG. 12(a), the detection unit 220 displays the cooperative field of view 131a on the display 131, analyzes the captured image captured by the outer camera 133, and determines whether the smartphone 500 is in a predetermined arrangement state based on the display content.

[0159] The associated visual field 131a is set to a part of the visual field range that the user 900 can see through the opening portion (lens portion) of the HMD 100 or the display area of ​​the display 131. Note that the entire visual field range that the user 900 can see through the HMD 100 may be set to the associated visual field 131a by changing the device configuration, setting values, etc.

[0160] The detection unit 220 determines that the predetermined arrangement state is present when a predetermined application icon 822 displayed on the display 531 of the smartphone 500 is within the linked visual field 131a as shown in Fig. 12(a). Generally, a plurality of icons 822, 823, and 824 are displayed on the display 531 of the smartphone 500 as shown in Fig. 12(a). The predetermined icon is, for example, the icon of an application selected by the user 900.

[0161] Whether or not the icon 822 of the application selected by the user 900 is within the linked visual field 131a is determined by analyzing the captured image 136. For example, the detection unit 220 determines the state and position where the finger of the user 900 is pointing by image analysis processing.

[0162] When the detection unit 220 determines that the predetermined arrangement state is established, it detects the icon 822 of the predetermined application as the linkage mark 820. Then, it specifies the application registered in the application list 250 in association with the detected linkage mark 820 as the linked application. If the application is not registered in the application list 250, it may be obtained from an external device such as a server and registered in the application list 250, as in the first embodiment.

[0163] In this embodiment, the application list 250 may store shape data of icons of applications installed in the device as the shape data 253 of the mark data 252.

[0164] In addition, as shown in FIG. 12(b), if the detection unit 220 determines that a specified icon 822 displayed on the display 531 of the smartphone 500 is not positioned within the linked field of view 131a, that is, that the HMD 100 and the smartphone 500 are not in a specified positional state, then it does not perform any subsequent detection processing, etc.

[0165] When the linking unit 240 detects the linking mark 820 , it generates detection notification data 280 a and transmits it to the smartphone 500 .

[0166] When the execution unit 230 receives a start instruction from the smartphone 500, the execution unit 230 starts and executes the linked app. Note that, also in this embodiment, when a linked app is specified, the linked app may be started in the HMD 100 without a start instruction from the smartphone 500.

[0167] Next, the smartphone 500 of this embodiment will be described.

[0168] FIG. 13 is a hardware configuration diagram of the smartphone 500. As shown in this figure, the smartphone 500 includes a main processor 511, a storage device 510, an operation reception device 520, an image processing device 530, a voice processing device 540, a sensor 550, a communication device 560, an extension interface unit 591, a timer 592, a vibration generating device 593, and an internal bus 512 connecting these. The storage device 510 also includes a RAM 513, a ROM 514, and a flash memory 515. The operation reception device 520 includes an operation key 521. The image processing device 530 includes a display 531, an out-camera 533, and an in-camera 534. The voice processing device 540 includes a speaker 541 and a microphone 543. The sensor 550 includes a GPS receiver 551, a gyro sensor 552, a geomagnetic sensor 553, an acceleration sensor 554, and a depth sensor 555. The communication device 560 includes a LAN communication interface 561, a telephone network communication interface 562, and a short-range wireless communication interface 563. These have the same functions as the components of the same names in the HMD 100.

[0169] However, the smartphone 500 has a larger display 531 area than the HMD 100. For this reason, the operation reception device 520 further includes a touch panel 523 having a touch sensor. For example, an input means of a touch pad type such as a capacitance type may be included. In this case, an approach or contact operation (touch operation) by a finger, a touch pen, or the like is detected as an operation input. The touch pad type allows the user 900 to set and input information that he or she wants to input, and to easily select and specify icons of apps and files on the display 531 by touch operation. The touch panel 523 is disposed on top of the display 531. Note that an input of an instruction may be received via a keyboard or the like connected to the extended interface unit 591.

[0170] The display 531 is configured with a liquid crystal panel or the like, and displays notification information to the user 900, such as the remaining battery capacity, various alarms, and the time, as well as icons of applications to be started and the execution status of the applications within the display 531. The operation reception device 520 is, for example, a touchpad-type input means such as a capacitance-type input means, and detects approach or contact operation (touch operation) by a finger, a touch pen, or the like as an operation input. The user 900 sets and inputs information that he or she wishes to input, and easily selects and designates icons of applications and files on the display 531 by touch operation.

[0171] The smartphone 500 of this embodiment displays an icon for accepting an instruction to start an application on the display 531. This icon is a mark for accepting an instruction to specify and start an application by an approach or contact operation (hereinafter referred to as touch) of the user 900 using a finger, a touch pen, or the like.

[0172] Next, a description will be given of the functions of the smartphone 500 according to the present embodiment.

[0173] As shown in this figure, the smartphone 500 of this embodiment includes a main control unit 570, a reception unit 571, a transmission / reception unit 572, a collaboration management unit 573, an application execution unit 574, an output unit 575, an icon data memory unit 577, an application program memory unit 576, and a collaboration group list 578.

[0174] The main control unit 570 controls the operation of each unit of the smartphone 500. In this embodiment, the main control unit 570 further determines the operation mode of the smartphone 500. In this embodiment, the main control unit 570 has two operation modes: a standalone operation mode in which the smartphone 500 alone executes an application, and a cooperative operation mode in which the smartphone 500 cooperates with the HMD 100 to execute an application.

[0175] For example, when the main control unit 570 receives the detection notification data 280a from the HMD 100 with which communication has been established, the main control unit 570 determines the operation mode to be the cooperative operation mode.

[0176] The reception unit 571 receives instructions from the user 900. In this embodiment, this is realized by the main processor 511 operating the operation reception device 520. In this embodiment, for example, various operation instructions such as an instruction to start an application are received.

[0177] The transmission / reception unit 572 transmits and receives data to and from an external device. In this embodiment, this is realized by the main processor 511 operating the communication device 560. In this embodiment, data is transmitted and received to and from the HMD 100 via the transmission / reception unit 572.

[0178] The cooperation management unit 573 performs cooperation operation when the operation mode of the smartphone 500 is determined to be the cooperation operation mode. The cooperation management unit 573 registers the mark ID 281, the device ID 282, and the address 283 of the received detection notification data 280a in the cooperation group list 578. Furthermore, when the cooperation management unit 573 receives the end notification data 280b from the HMD 100 registered in the cooperation group list 578, the cooperation management unit 573 deletes the data of the corresponding device. In this way, the cooperation group is managed. The configuration of the cooperation group list 578 is the same as that of the cooperation group list 260 in the first embodiment.

[0179] The cooperation management unit 573 also controls the cooperation of cooperation apps by the HMD 100 that is a member of each cooperation group. For example, it makes it easy to see which apps are running by displaying a mark indicating that the app is in use on the icon of the app being used or by displaying the number of participants. This allows people who join later to check the apps that participants are using on the display and then select and participate.

[0180] In the independent operation mode, the application execution unit 574 executes an application on the smartphone 500. In the independent operation mode, when an instruction to start an application is received from the user 900, the application is started and executed on the smartphone 500. The program of the application to be executed is stored in the application program storage unit 576.

[0181] The output unit 575 outputs the execution result of the application executed by the application executing unit 574. In this embodiment, the execution result is output to the display 531 and the speaker 541, for example.

[0182] Application programs are stored in application program storage unit 576. Icons for receiving instructions to execute each application program are stored in icon data storage unit 577. The icons are stored in association with the applications.

[0183] The above functions are realized by main processor 511 loading a program stored in ROM 514 into RAM 513 and executing it. An application program storage unit 576 and an icon data storage unit 577 are stored in storage device 510.

[0184] [Linkage processing] The flow of the linking process in this embodiment will be described with reference to FIG.

[0185] First, communication is established between the HMD 100 and the smartphone 500 (steps S3101 and S3201). Here, communication settings are made between the smartphone 500 and the multiple HMDs 100 to enable communication. For example, when the HMD 100 enters the communication range of the smartphone 500, communication is automatically or manually established between the two.

[0186] After that, the smartphone 500 displays the icon 822 (step S3202). Note that here, a normal screen on which a large number of icons are displayed is displayed. The icon 822 to be used as the linkage mark 820 is selected and instructed by the user 900 as described above.

[0187] The subsequent processing is substantially the same as the cooperation processing with the server 400 in the second embodiment. That is, when the detection unit 220 of the HMD 100 detects the cooperation mark 820 (step S1101), the detection unit 220 of the HMD 100 identifies an application registered in association with the detected cooperation mark 820 as a cooperation application (step S1102).

[0188] The link unit 240 of the HMD 100 generates detection notification data 280a and transmits it to the smartphone 500 (step S2104).

[0189] When the cooperation management unit 573 of the smartphone 500 receives the detection notification data 280a (step S3203), the cooperation management unit 573 stores the data of the HMD 100 that is the transmission source in the cooperation group list 578, thereby updating the cooperation group list 578 (step S3204).

[0190] Then, when the cooperation management unit 573 finishes updating, it returns data (start-up instruction) indicating that the process of joining the cooperation group is completed and instructing startup to the transmission source HMD 100 (step S3205). In addition, it transmits the cooperation group list to all HMDs 100 registered in the cooperation group list 578.

[0191] When the execution unit 230 of the HMD 100 receives the start instruction, the execution unit 230 of the HMD 100 starts and executes the linked application (step S1105). As in the first embodiment, the execution unit 230 of the HMD 100 continues to execute the executed application until it receives an instruction to end the executed application (step S1106).

[0192] During execution of the linked application, in this embodiment, the execution unit 230 transmits and receives data required for linked execution to and from other HMDs 100 registered in the linked group list 578 (step S2108).

[0193] When the execution unit 230 of the HMD 100 receives an instruction to end the linked app from the user 900, the execution unit 230 ends the linked app, and the link unit 240 generates end notification data 280b, transmits it to the smartphone 500 (step S2109), and ends the process. On the other hand, the link management unit 573 of the smartphone 500 that received the end notification data 280b (step S3207) updates the linked group list 578 by deleting the HMD 100 that sent the data from the linked group list 578 (step S3208). Furthermore, the updated linked group list 578 is transmitted to each HMD 100 registered in the updated linked group list 578.

[0194] The same is true for other HMDs 100, and when the collaboration mark 810 is detected, the detection notification data 280a is transmitted to the smartphone 500. When the smartphone 500 receives the detection notification data 280a or the end notification data 280b, the smartphone 500 updates the collaboration group list 578 as needed, and returns the updated collaboration group list 578.

[0195] In this embodiment, if there is a time difference between the determination of whether or not the icon 822 is in a predetermined arrangement state and the touch of the icon 822 by the user 900, the HMD 100 that is not looking at the icon 822 at that time may join the cooperation group, and the HMD 100 that is looking at the icon 822 at that time may not join the cooperation group. For this reason, the storage interval (the interval for updating the cooperation group list 578) may be shortened so that no time difference occurs between the two. Also, the icon touch may be recognized by the outer camera 133 of the HMD 100. In this case, since the determination of the time of the touch moment may change depending on the viewing angle, the determination period for the icon touch may be given a certain time range.

[0196] As described above, the information processing system 702 of the present embodiment executes an application in cooperation with the HMDs 100, which are multiple information processing terminals. Each HMD 100 includes a detection unit 220 that detects a cooperation mark 820 displayed on the display 531 of the smartphone 500, and an execution unit 230 that executes an application associated with the detected cooperation mark 820 as a cooperation app in cooperation with another HMD 100 that has detected the same cooperation mark 820. Furthermore, the information processing system 702 of the present embodiment further includes a smartphone 500 that includes a cooperation management unit 573 that manages a plurality of HMDs 100 that have detected the same cooperation mark 820 as members of a cooperation group that cooperates to execute a cooperation app, and the execution unit 230 transmits data obtained by processing the cooperation app to the other HMDs 100 and receives data obtained by processing the cooperation app from the other HMDs 100.

[0197] According to this embodiment, as in the first embodiment, application linkage can be realized simply, safely, and conveniently. Furthermore, according to this embodiment, a linkage mark 820 is displayed on the display 531 of the smartphone 500. Therefore, when the HMD 100 and the smartphone 500 are in a predetermined arrangement state, such as an icon 822 displayed on the display 531 of the smartphone 500 being visible within the linkage visual field 131a of the HMD 100, a linkage group can be formed between the HMD 100 having the same arrangement state as the smartphone 500, and a linkage app can be executed in cooperation with the smartphone 500. Therefore, according to this embodiment, it is possible to easily and conveniently identify a linkage app and identify a linking HMD 100.

[0198] Furthermore, according to this embodiment, the coordination mark 820 is displayed on the display 531 of the smartphone 500. Unlike coordination marks that are physically printed and cannot be easily changed, in this embodiment, a large number of coordination marks can be displayed within a limited area while being switched by scrolling or the like. This increases the number and options of usable coordination marks. In addition, the arrangement position of the coordination mark 820 can be moved. This allows the user 900 of the HMD 100 to recognize the coordination mark 820 at a position that is easy to see, and the user 900 can easily view the coordination mark 820. In other words, it becomes easier for the HMD 100 to detect the coordination mark 820.

[0199] In addition, the smartphone 500 can distinguish between the cooperative operation mode and the independent operation mode. Therefore, when the user 900 selects the icon 822 displayed on the display 531, it can automatically determine whether to run the corresponding app on the smartphone 500 or to control the cooperative operation.

[0200] <Variation 6> In the above embodiment, the icon 822 of the smartphone 500 is viewed through the field of view of multiple HMDs 100. For this reason, when multiple icons are displayed on the display 531 of the smartphone 500, the user 900 needs to constantly select the icon 822 while the HMD 100 to be linked detects it as the linkage mark 820. However, the present invention is not limited to this method.

[0201] For example, when the smartphone 500 first receives the detection notification data 280a and generates the collaboration group list 578, the smartphone 500 may display only the icon 822 selected by the user 900 on the display 531. After that, the icon 822 functions as the collaboration mark 820 without a selection instruction from the user 900.

[0202] As a result, a specific icon 822 is displayed on the display 531 as the collaboration mark 820 for a predetermined period of time, making it easier for the HMD 100 that wants to participate in the collaboration group to participate. On the other hand, even if this configuration is adopted, the user 900 who does not want to participate does not need to look at the display of the smartphone 500.

[0203] <Variation 7> In the above embodiment, the smartphone 500 transmits the start-up instruction to the HMD 100 that has transmitted the detection notification data 280a every time the smartphone 500 receives the detection notification data 280a and registers the data in the cooperation group list 578. However, the present invention is not limited to this. For example, the smartphone 500 may transmit the start-up instruction to the HMD 100 that has transmitted the detection notification data 280a all at once.

[0204] In this case, for example, the cooperation management unit 573 registers each detection notification data 280a received in the cooperation group list 578. After that, when a start instruction is received by touching the icon 822 set as the cooperation mark 820, the cooperation management unit 573 transmits the start instruction to all HMDs 100 registered in the cooperation group list 578. When each HMD 100 receives the start instruction from the smartphone 500, it starts the cooperation app in response to the start instruction.

[0205] <Variation 8> Also in this embodiment, similarly to the modified example of the second embodiment, the smartphone 500 may control the execution of the linked app. In this case, the link management unit 573 transmits data received from each HMD 100 that executes the linked app to the other HMD 100.

[0206] In this case, the cooperation group list 578 may be managed only in the smartphone 500. In other words, the cooperation management unit 573 does not need to transmit the cooperation group list 578 to the HMD 100 in step S3205 or step S3208.

[0207] In each HMD 100, while the linked app is being executed, the execution unit 230 transmits data required for the linked execution to the smartphone 500. In addition, in the smartphone 500, the linkage management unit 573 refers to the linkage group list 578, and when receiving data from an HMD 100 in the linked group, processes the data and transmits it to the other HMD 100 in the linked group.

[0208] <Variation 9> Furthermore, the smartphone 500 may not only mediate the data of the linked application but also execute the linked application by itself. In this case, the smartphone 500 registers its own device ID and address in the linked group list 578. Then, the application execution unit 574 executes the linked application even in the linked operation mode.

[0209] <Modification 10> In the above embodiment, whether or not the smartphone 500 and the HMD 100 are in a predetermined arrangement state is determined using an image captured by the outer camera 133 of the HMD 100. However, this is not limited to this. For example, information from the sensor 550 of the smartphone 500 may be used to determine whether or not the smartphone 500 and the HMD 100 are in a predetermined arrangement state.

[0210] For example, the attitude and position of the HMD 100 are detected using the acceleration sensor 154, the gyro sensor 152, the geomagnetic sensor 153, etc., and the relationship between the attitude of the HMD 100 and the attitude and position of the smartphone 500 is determined. The attitude and position of the smartphone 500 are calculated by the acceleration sensor 554, the gyro sensor 552, and the geomagnetic sensor 553 provided in the smartphone 500. The distance between the smartphone 500 and the HMD 100 is acquired by the depth sensor 155, etc.

[0211] <Modification 11> In the above embodiment, when an icon 822 displayed on the display 531 of the smartphone 500 or the like is present within the predetermined linkage visual field 131a and is selected by the user 900, the predetermined arrangement state is determined, and the icon 822 is used as the linkage mark 820. However, this is not limited to this. For example, the icon 822 to be used as the linkage mark 820 may be selected via the HMD 100 from among a plurality of icons displayed on the display 531 of the smartphone 500.

[0212] <Modification 12> Furthermore, the link mark 820 displayed on the display 531 of the smartphone 500 does not have to be the icon 822 of the link app. For example, it may be an icon indicating file data. In this case, an app that plays the file data is stored in association with the mark data as the app list 250. This allows the app to be started on multiple HMDs 100, and the file data to be shared.

[0213] In addition, file data may be playable not only by the same application but also by different applications. Therefore, different applications may be associated with the application list 250 in each HMD 100. This allows file data to be played back in a coordinated manner even between HMDs 100 equipped with different applications.

[0214] Furthermore, when the file data is an AR object, the coordinates of the AR object may be shared among the HMDs 100 by aligning the coordinate systems of the HMDs 100 with the coordination mark 820 as a reference.

[0215] <Modification 13> Furthermore, a specific icon 822 displayed on the display 531 of the smartphone 500 may be displayed in a manner that makes it clear (linkage display) while it is being used as the linkage mark 820. Being used as the linkage mark 820 means that a linkage app associated with the icon 822 is being executed by a plurality of HMDs 100 in a linked manner.

[0216] The link display may be, for example, by changing the display color of the icon 822, blinking it, attaching a small mark such as a dot 825 to the icon 822 as shown in Fig. 14(b), displaying the number of linked HMDs, etc. This allows the user to know which icon corresponds to which linked app is being executed in linkage, even when icons of multiple apps are displayed.

[0217] In this way, by displaying icons in a linked manner, it becomes easier to distinguish which apps are being executed in a linked manner while the linked apps are being executed in a linked manner. This makes it easier for a mid-game joiner to distinguish which icons they should recognize.

[0218] <Modification 14> In each of the above embodiments and modifications, when a change occurs in the constituent members of a cooperation group, such as when a member is registered in the cooperation group list or, conversely, deleted from the cooperation group list, each HMD 100 may be configured to notify the user 900. The notification is performed, for example, by displaying on the display 131, outputting sound from the speaker 141, vibrating the vibration generating device 173, etc. This can improve the convenience for the user 900.

[0219] <Variation 15> In addition, when another HMD 100 joins the linked group after the linked app has been linked and execution has begun, the HMD 100 may be configured to join with permission from all HMDs 100 already participating in the linked group or from the representative HMD 100.

[0220] For example, when an HMD 100 that wishes to participate transmits detection notification data 280a, each HMD 100 or the representative HMD 100 checks whether participation is permitted. For example, the check is made by authentication processing. Then, only if permitted, the HMD 100 is added to the linked group list. Then, a notification of participation permission is transmitted to the HMD 100 that transmitted the data.

[0221] <Modification 16> The information terminal that executes the linked app in a linked manner is not limited to the HMD 100. For example, it may be a smartphone 500, a personal computer, a tablet, or the like. These information terminals may be mixed. It is sufficient that the information processing terminal has at least a configuration capable of determining that the link marks 810 and 820 have been detected and a configuration capable of executing the linked app.

[0222] <Modification 17> Moreover, the cooperation mark 820 does not have to be a mark in the real space. For example, it may be a mark in a VR (Virtual Reality) space. For example, in the VR space, when the avatars, which are the avatars of the respective users 900, recognize the same cooperation mark, the avatars cooperate to execute the cooperation app corresponding to the recognized cooperation mark. The recognition in the VR space is realized, for example, by having the avatar touch (selection instruction) the cooperation mark and the cooperation mark in the VR space. As a result, even if many users (avatars) exist in the VR space, the app can be executed in cooperation only by the avatars of some of the users who have made a selection instruction on the cooperation mark.

[0223] <Application Examples> In the third embodiment, the device that displays the cooperation mark 820 is not limited to the display 531 of the smartphone 500. For example, it may be a screen that projects an image.

[0224] For example, all users 900 gathered in a conference room wear HMDs 100 and view the cooperation mark displayed on a screen (display) in the conference room through their field of vision. The HMDs 100 detect the cooperation mark. Each HMD 100 is associated with a cooperation app. Each HMD 100 identifies and starts the corresponding cooperation app. Start notification data is also transmitted and received, and the cooperation group is managed.

[0225] As a result, a collaborative group for the conference is automatically formed by the HMDs 100 of the users 900 who have gathered in the conference room, and collaborative applications can be executed in a collaborative manner.

[0226] For example, by using a collaborative app that has various meeting processing functions such as sharing materials only within the collaborative meeting group, recording participants' comments, and creating minutes, the meeting can be run more smoothly and its efficiency can be easily increased.

[0227] Also, by displaying a linkage mark on one side of the screen during a meeting, an HMD 100 that joins midway can start a linkage app and participate in the meeting just by detecting the linkage mark. In particular, in meetings, the members who participate often change each time. However, even in such a situation, according to this example, a linkage group can be automatically formed just by displaying an icon of a linkage app on the screen, and the meeting can be smoothly managed. This further improves convenience.

[0228] The second embodiment can also be applied to a remote conference. For example, in a situation where there are multiple conference locations. In this case, the same cooperation mark 810 is provided in advance at each conference location. The HMD 100 that wishes to participate can automatically start a cooperation app by simply detecting the cooperation mark 810, and a cooperation group can be formed between the HMDs 100 in remote locations.

[0229] A linked application can be executed between the HMDs 100 registered as a linked group by communication over a network such as a LAN via the server 400. By making the linked application an application that realizes various functions of a conference, a remote conference can be easily carried out.

[0230] In this case, the linking application may be started simultaneously for each conference location or each icon installation location by a trigger such as touching an icon, or may be started at each timing of each HMD 100. After forming a linking group for each conference location in each conference location, the server 400 may integrate the linking groups of all conference locations. In this case, the server 400 may be configured to connect to the server 400 via the HMD 100 representing the linking group of each conference location.

[0231] The present invention is not limited to the above-described embodiments and modifications, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0232] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0233] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0234] 100: HMD, 101: HMD, 102: HMD, 103: HMD, 104: HMD, 110: storage device, 111: main processor, 112: system bus, 113: RAM, 114: ROM, 115: flash memory, 120: operation reception device, 121: operation key, 130: image processing device, 131: display, 131a: linked visual field, 133: outer camera, 134: inner camera, 136: captured image , 137: area, 140: audio processing device, 141: speaker, 143: microphone, 150: sensor, 151: GPS receiver, 152: gyro sensor, 153: geomagnetic sensor, 154: acceleration sensor, 155: depth sensor, 160: communication device, 161: LAN communication I / F, 162: telephone network communication I / F, 163: short-range wireless communication I / F, 171: expansion interface unit, 172: timer, 173: vibration generator, 210: Main control unit, 220: Detection unit, 230: Execution unit, 240: Collaboration unit, 250: Application list, 251: Application ID, 252: Mark data, 253: Shape data, 254: Mark ID, 260: Collaboration group list, 261: Mark ID storage location, 262: Device ID, 263: Address, 270: Processing data, 272: Mark data, 280: Notification data, 280a: Detection notification data, 280b: End notification data, 281: Mark ID, 282: Device ID, 283: Address, 284: Application ID, 285: Status, 300: Network, 400: Server, 401: CPU, 402: Memory, 403: Storage device, 404: Communication interface, 410: Main control unit, 420: Transmitter / receiver, 430: Collaboration control unit, 440: Collaboration management unit, 460: Collaboration group list, 461: Mark ID storage location, 462: Device ID, 463: Address, 500: smartphone, 510: storage device, 511: main processor, 512: internal bus, 513: RAM, 514: ROM, 515: flash memory, 520: operation reception device, 521: operation key, 523: touch panel, 530: image processing device, 531: display, 533: outer camera, 534: inner camera, 540: audio processing device, 541: speaker, 543: microphone, 550: sensor, 551: GPS receiving device, 552: gyro sensor, 553: geomagnetic sensor, 554: Acceleration sensor, 555: depth sensor, 560: communication device, 561: LAN communication interface, 562: telephone network communication interface, 563: short-range wireless communication interface, 570: main control unit, 571: reception unit, 572: transmission / reception unit, 573: cooperation management unit, 574: application execution unit, 575: output unit, 576: application program storage unit, 577: icon data storage unit, 578: cooperation group list, 591: expansion interface unit, 592: timer, 593: vibration generator, 700: information processing system, 701: information processing system, 702: information processing system, 810: Collaboration mark, 820: Collaboration mark, 822: Icon, 823: Icon, 824: Icon, 825: Dot, 900: user, 901: user, 902: user, 903: user, 904: user, 912: wireless communication, 913: wireless communication, 914: wireless communication, 923: wireless communication, 924: wireless communication, 934: wireless communication, 941: line of sight, 942: line of sight, 943: line of sight, 944: line of sight

Claims

1. An information processing terminal for executing an application, A communication unit for transmitting and receiving information between the device and other terminals; A detection unit that detects a predetermined linkage mark; a coordination unit that, when the detection unit detects the coordination mark, transmits a notification indicating that the coordination mark has been detected to the other terminal via the communication unit, and, when the communication unit receives a notification indicating that the coordination mark has been detected from the other terminal, manages the other terminal as a member terminal of a coordination group that executes an application in coordination with the other terminal; an execution unit that executes an application associated with the cooperation mark in cooperation with the other terminals that are managed by the cooperation unit as constituent terminals of the cooperation group; The information processing terminal, wherein the detection unit detects a mark of a predetermined application as the linkage mark when the information processing terminal determines that the information processing terminal is in a predetermined arrangement state.

2. 2. The information processing terminal according to claim 1, the information processing terminal further includes a camera that captures an image of the surroundings of the information processing terminal; The information processing terminal, wherein the detection unit determines whether or not the information processing terminal is in the predetermined arrangement state based on an image of the surroundings acquired by the camera.

3. 2. The information processing terminal according to claim 1, the information processing terminal further includes a camera that captures an image of the surroundings of the information processing terminal; The information processing terminal, wherein the detection unit detects the predetermined link mark based on an image of the surroundings acquired by the camera.

4. 2. The information processing terminal according to claim 1, the information processing terminal further includes a reception unit that receives an instruction from a user of the information processing terminal; An information processing terminal characterized in that, after execution of an application corresponding to the link mark, when the receiving unit receives an instruction to terminate the application, the execution unit terminates the execution of the application and the communication unit sends a termination notification to the other terminal.

5. 2. The information processing terminal according to claim 1, the information processing terminal further includes a memory that stores the linkage mark and a list of applications associated with the linkage mark; The communication unit transmits and receives information to and from a server on a network, When a mark not stored in the list is detected by the detection unit, the information processing terminal transmits inquiry data to the server by the communication unit.

6. 6. An information processing terminal according to claim 1, The other terminals are terminals located in the vicinity of the information processing terminal.

7. 6. An information processing terminal according to claim 1, The information processing terminal is characterized in that the link mark is an icon for calling up a start function of an application.

Citation Information

Patent Citations

  • Electronic device, method of controlling the same, and program

    JP2012227814A

  • Terminal device program

    JP2019219944A

  • System and method for enabling a mobile chat session

    US20120244891A1

  • Determination device, determination method, and determination program

    JP2016174253A

  • Information processing system, information processing device, information processing method and information processing program

    JP2017142693A