Image sharing system and image sharing method

Abstract

The image sharing system includes a first portable information terminal and a second portable information terminal. The first portable information terminal includes: a first display; a first communication device for transmitting and receiving data to and from the second portable information terminal; and a first processor that controls the first display and the first communication device, in which the first processor performs visual sharing control in accordance with a selection operation by a user of the first portable information terminal or a user of the second portable information terminal, a display mode of an individual display object composed of a virtual object serving as a visual sharing target is switched from a video viewed from a viewpoint position of a first portable information terminal to a video viewed from a viewpoint direction of a second portable information terminal based on data received via a first communication device. The first display displays the individual display objects using an image viewed from the viewpoint direction of the second portable information terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to an image sharing system and an image sharing method, and particularly relates to an image sharing technology in a virtual space between portable information terminals. BACKGROUND

[0002] Patent Document 1 describes the following content, "When users A, B communicate while facing each other and watching the images of the other party, object display regions 42A, 42B are provided in front of the avatars 41A, 41B of the respective users A, B, and objects 43A, 43B are displayed in the respective object display regions, and in the case where the user A or B performs an operation such as movement, rotation, stretching, deformation, etc. on the object 43A or 43B, the operation is also performed on the other object (selected from the abstract)".

[0003] PRIOR ART DOCUMENT

[0004] PATENT DOCUMENT

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-22958 SUMMARY

[0006] PROBLEM TO BE SOLVED BY THE INVENTION

[0007] In the prior art, for an object being discussed, a copy of the object corresponding to the number of users who want to share the object needs to be displayed, and the more the number of sharing users increases, the more the number of displayed objects increases, and there is a problem that a calculation load increases and display congestion occurs in a virtual space.

[0008] The present application has been made in view of the above circumstances, and has an object to provide an image sharing system and an image sharing method that reduce display congestion when an object is shared and displayed among a plurality of users.

[0009] MEANS OF SOLVING THE PROBLEM

[0010] To solve the above problems, the present application has the structure described in the claimed technical solution. As one example, the present application is an image sharing system including a first portable information terminal and a second portable information terminal, the first portable information terminal including: a first display; a first communication device that performs data transmission and reception with the second portable information terminal; and a first processor that controls the first display and the first communication device, wherein the first processor performs visual sharing control in accordance with a selection operation of a user of the first portable information terminal or a user of the second portable information terminal, in which the display mode of an individual display object composed of a virtual object selected as a visual sharing target is switched from an image viewed from a viewpoint position of the first portable information terminal to an image viewed from a viewpoint direction of the second portable information terminal based on the data received via the first communication device, and the first display displays the individual display object using the image viewed from the viewpoint direction of the second portable information terminal.

[0011] In addition, the present application is an image sharing method that performs the following steps: a first step in which a portable information terminal performs visual sharing control to switch the display mode of a virtual object selected as a visual sharing target among a plurality of users from an image viewed from a viewpoint position of the portable information terminal to an image viewed from a viewpoint direction of another portable information terminal; and a second step in which the portable information terminal displays the virtual object using the image viewed from the viewpoint direction of the other portable information terminal.

[0012] Effects of the Invention

[0013] According to the present application, in order to make the virtual object targeted among users appear to be the same, it is not necessary to generate a copy, but to switch the display of the virtual object targeted to be viewed from the same direction. Thereby, it is possible to obtain the effect of suppressing an increase in computational load, and not causing display congestion in the virtual space. The above-mentioned objects, features, effects, and the like will be described in the following embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic configuration diagram of an image sharing system of the first embodiment.

[0015] Figure 2 is a hardware configuration diagram of an HMD.

[0016] Figure 3 is a hardware configuration diagram of a smartphone.

[0017] Figure 4 is a hardware configuration diagram of a server.

[0018] Figure 5 is a functional block diagram of an image sharing program.

[0019] Figure 6 is a view that represents an image of a virtual space viewed from a viewpoint of a terminal in general.

[0020] Figure 7 is a view that represents an image of a virtual space viewed from a viewpoint of each terminal at a time point when visual sharing of a selected virtual object is started.

[0021] Figure 8 is a view that represents a state of a virtual space in visual sharing.

[0022] Figure 9 is a view that represents an example of a state of a virtual space after visual sharing is ended.

[0023] Figure 10 is a view that represents a coordinate system of an individual display object.

[0024] Figure 11 is a view that represents a position and an orientation of an individual display object coordinate system with respect to a virtual space coordinate system.

[0025] Figure 12 is a view that represents a position relationship between two terminal viewpoints and a virtual object in a state of Figure 6 .

[0026] Figure 13 is a view that represents a position relationship between two terminal viewpoints and a virtual object at a time when visual sharing of Figure 7 is started.

[0027] Figure 14 is a view that represents a display reference configuration in a host terminal at a time when visual sharing is started.

[0028] Figure 15 is a view that represents a display reference configuration in a slave terminal at a time when visual sharing is started.

[0029] Figure 16 is a view that represents a relationship between a horizontal reference direction of a host terminal and a horizontal reference direction of a slave terminal.

[0030] Figure 17 is a view that represents an example of a host terminal marker.

[0031] Figure 18 is a view that represents another example of a host terminal marker.

[0032] Figure 19 is a flowchart that represents a processing flow of an image sharing system of the present embodiment, and represents a manner in which display control of an individual display object is performed in a portable information terminal.

[0033] Figure 20Ais a flowchart showing a processing flow of the image sharing system of the present embodiment, and shows a manner in which display control of individual display objects is performed in the server.

[0034] Figure 20B is a flowchart showing a processing flow of the image sharing system of the present embodiment, and shows a manner in which display control of individual display objects is performed in the server.

[0035] Figure 21 is a diagram showing a display example of Modification 3 of the first embodiment.

[0036] Figure 22 is a diagram showing a display example of Modification 4 of the first embodiment.

[0037] Figure 23 is a diagram showing a display example of the second embodiment.

[0038] Figure 24 is a diagram showing a display example of Modification 1 of the second embodiment.

[0039] Figure 25 is a diagram showing a display example of Modification 2 of the second embodiment.

[0040] Figure 26 is a diagram showing a display example of Modification 3 of the second embodiment.

[0041] Figure 27 is a diagram showing a display example of Modification 4 of the second embodiment.

[0042] Figure 28 is a diagram showing a display example of Modification 5 of the second embodiment.

[0043] Figure 29 is a diagram showing a display example of Modification 6 of the second embodiment. DETAILED DESCRIPTION

[0044] In the present application, it is possible to view a virtual object in a manner in which the virtual object is presented (visual effect) from a viewpoint direction of another user located at a different position, i.e., it is possible to perform visual sharing, between a plurality of portable information terminals, and it is expected that this will promote mutual understanding and improve communication. Therefore, it is expected that the present application can improve technology for labor-intensive industries that require work assistance and logistics assistance, and thus it is expected to contribute to 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (achieve higher levels of economic productivity through diversification, technological upgrading, and innovation, including focusing on developing high-value-added and labor-intensive industries).

[0045] In the image sharing system of the present embodiment, the first portable information terminal and the second portable information terminal are connected via a communication line, and a visual sharing control is performed in accordance with a selection operation by a user of the first portable information terminal or a user of the second portable information terminal, so that a display mode of an individual display object constituted by a virtual object that is a visual sharing target is switched from an image viewed from a viewpoint position of the first portable information terminal to an image viewed from a viewpoint direction of the second portable information terminal based on data received via the first communication device, and the first display displays the individual display object using the image viewed from the viewpoint direction of the second portable information terminal.

[0046] Embodiments of the present application will be described below with reference to the accompanying drawings. In all the drawings used to describe the embodiments, the same reference numerals are applied to the same components, and repetitive descriptions thereof will be omitted.

[0047] <First Embodiment>

[0048] In the first embodiment, in accordance with a difference between a line-of-sight direction of the first portable information terminal and a line-of-sight direction of the second portable information terminal as viewed from the individual display object, a position and an attitude of the individual display object in the virtual space as viewed from the line-of-sight position of the first portable information terminal are transformed into a position and an attitude of the individual display object in the virtual space as viewed from the line-of-sight position of the second portable information terminal, and an image of the individual display object as viewed from the first portable information terminal is switched to an image of the individual display object as viewed from the second portable information terminal, whereby a visual sharing control is performed.

[0049] Figure 1 is a schematic configuration diagram of the image sharing system of the first embodiment.

[0050] As shown in Figure 1 , a user A wears an HMD (Head Mounted Display) 1A as the portable information terminal 1 and operates a smartphone 1C as the portable information terminal 1. Similarly, a user B wears an HMD 1B as the portable information terminal 1 and operates the smartphone 1C as the portable information terminal 1.

[0051] The smartphone 1C is connected to the network 9 via an access point 23A, and the smartphone 1D is connected to the network 9 via an access point 23B, and they are communicatively connected to a server 3 connected to the network 9. In the case where the portable information terminals 1 are located in the same place, the same access point can be used.

[0052] A close-range wireless connection is employed between the HMD 1A and the smartphone 1C, and a close-range wireless connection is employed between the HMD 1B and the smartphone 1D. In addition, the HMDs 1A, 1B can also be communicatively connected to the access points 23A, 23B directly.

[0053] The image sharing system 100 directly communicates between the HMDs 1A and 1B, or indirectly via the network 9, to configure the same virtual object 6, 7, 8 in the same virtual space for the users A, B to view.

[0054] Figure 2 is a hardware configuration diagram of the HMDs 1A, 1B.

[0055] Since the HMDs 1A and 1B are structurally the same, in the following description, the suffixes of the reference numerals are distinguished by A, B for the constituent elements common to the HMDs 1A and 1B.

[0056] In the HMDs 1A and 1B, the outside cameras 111A, 111B, the distance measuring sensors 113A, 113B, the position and posture sensors 135A, 135B, the geomagnetic sensors 117A, 117B, the positioning sensors 118A, 118B, the RTCs (real-time clocks) 114A, 114B, the displays 119A, 119B, the microphones 121A, 121B, the speakers 122A, 122B, the processors 125A, 125B, the memories 128A, 128B, the wireless communication devices 129A, 129B, and the network communication devices 120A, 120B are connected to each other via buses 140A, 140B connecting the constituent elements. The network communication devices 120A, 120B are connected to antennas 123A, 123B that transmit and receive network communication signals. The displays 119A, 119B, the speakers 122A, 122B correspond to output devices, and the microphones 121A, 121B correspond to input devices.

[0057] The position and posture sensors 135A, 135B are used to measure the position and posture of the HMDs 1A, 1B. Here, the posture refers to the three-dimensional rotational position of the HMDs 1A, 1B in the real space in which the HMDs 1A, 1B are located. Details will be described later. As a specific configuration example of the position and posture sensors 135A, 135B, there are position and posture cameras 136A, 136B (Position AND Posture camera: hereinafter, simply referred to as "PP camera"), position and posture distance measuring sensors 137A, 137B (Position AND Posture distance measuring sensor: hereinafter, simply referred to as "PP distance measuring sensor"), acceleration sensors 115A, 115B, and gyro sensors 116A, 116B. The PP cameras 136A, 136B can be the same cameras as the outside cameras 111A, 111B. In addition, the PP distance measuring sensors 137A, 137B can be the same sensors as the distance measuring sensors 113A, 113B.

[0058] In the case of measuring the position and the attitude using the PP cameras 136A, 136B and the PP distance sensors 137A, 137B, a feature point present in the outside world is used. The feature point is used in plural, and can be a feature point of a photographic subject, a feature point in a room, or a feature point of a building in the outside world, for example. As a preparation process, the relative positional relationship of the feature point is measured by measuring the direction and the distance of the feature point. This preparation can be omitted if the coordinate value of the feature point in the outside coordinate system is known. Then, the change in the direction and the distance of the feature point as viewed from the HMDs 1A, 1B is measured, and thus the change in the position and the attitude of the HMDs 1A, 1B is measured. In the case of measuring the distance using the stereo camera method, only the PP cameras 136A, 136B can be provided. In the case of measuring the attitude using the PP cameras 136A, 136B and the PP distance sensors 137A, 137B if the value of the feature point in the outside coordinate system is known and the relationship between the outside coordinate system and the vertical direction is known, the relative positional relationship of the housing attitude of the HMDs 1A, 1B and the vertical direction can be grasped.

[0059] In the case of measuring the position and the attitude using the acceleration sensors 115A, 115B and the gyro sensors 116A, 116B, the measurement is performed by integrating the acceleration and the angular acceleration. Thus, the change from the reference position and the reference attitude is found. In addition, by measuring the gravitational acceleration vector using the acceleration sensors 115A, 115B, the relative positional relationship of the housing attitude of the HMDs 1A, 1B and the vertical direction can be grasped.

[0060] In measuring the position and the attitude of the HMDs 1A, 1B, both of the measurement values obtained using the PP cameras 136A, 136B and the PP distance sensors 137A, 137B and the measurement values obtained using the acceleration sensors 115A, 115B and the gyro sensors 116A, 116B can be referred to, or the measurement values of other types of sensors can be used.

[0061] The positioning sensors 118A, 118B are used to measure the position of the HMDs 1A, 1B in a room or in the outside world. This includes positioning with lower accuracy than the position attitude sensors 135A, 135B which measure the position of the terminal housing. For example, this is used to obtain position information which is used as an index when searching for a library image related to a photographic subject. As a specific example, a beacon signal is used in a room, a position marker is used, or a GPS (Global Positioning System) receiver which uses a satellite signal is used in the outside world.

[0062] The network communication machines 120A, 120B are communication interfaces that perform wireless communication at least between the HMDs 1A, 1B and the access points 23A, 23B by close proximity wireless communication, wireless LAN, or base station communication, and include communication processing circuits corresponding to prescribed various communication interfaces, and are connected to the antennas 123A, 123B. Further, the network communication machines 120A, 120B receive biological information, image data, control signals, and the like. In addition, the close proximity wireless communication is performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark) or the like wireless LAN. In addition, the base station communication can use LTE (Long-Term Evolution, registered trademark) system, LTE-Advanced system, mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark) system, WiMAX2 system, or the like long distance wireless communication.

[0063] The displays 119A, 119B are disposed in front of the eyes of the users A, B wearing the HMDs 1A, 1B. The processors 125A, 125B control the actions of the HMDs 1A, 1B. In particular, the images of the real space information captured by the outside cameras 111A, 111B and the captured images received from the access points 23A, 23B are corrected and displayed on the displays 119A, 119B.

[0064] The processors 125A, 125B are constituted by, for example, CPUs.

[0065] The memories 128A, 128B are constituted by flash memories, nonvolatile memories. The memories 128A, 128B store programs 126A, 126B such as an OS (Operating System) and / or an action control application, and data 127A, 127B used for the processors 125A, 125B.

[0066] The processors 125A, 125B load the programs 126A, 126B into the memories 128A, 128B and execute them, and read the data 127A, 127B as necessary for the execution processing of the programs 126A, 126B.

[0067] Figure 3 is a hardware configuration diagram of a smartphone.

[0068] Parts having the same names as the constituent element names of the HMD have substantially the same functions, so detailed description is omitted in the absence of particular differences.

[0069] In the smartphones 1C, 1D, the rear cameras 211A, 211B, the front cameras 212A, 212B, the distance sensors 213A, 213B, the position and posture sensors 235A, 235B, the geomagnetic sensors 217A, 217B, the positioning sensors 218A, 218B, the RTCs 214A, 214B, the displays 219A, 219B, the microphones 221A, 221B, the speakers 222A, 222B, the processors 225A, 225B, the memories 228A, 228B, the network communication machines 220A, 220B, the telephone network communication machines 231A, 231B, and the wireless communication machines 229A, 229B are connected to each other via buses 240A, 240B connecting the respective constituent elements. The network communication machines 220A, 220B are connected to antennas 223A, 223B that transmit and receive network communication signals.

[0070] The position and posture sensors 235A, 235B have the same specific structures as in the case of the HMD, and there are the PP cameras 236A, 236B, the PP distance sensors 237A, 237B, the acceleration sensors 215A, 215B, and the gyro sensors 216A, 216B. The PP cameras 236A, 236B can be the same cameras as the rear cameras 211A, 211B. In addition, the PP distance sensors 237A, 237B can be the same sensors as the distance sensors 213A, 213B.

[0071] The memories 228A, 228B store the programs 226A, 226B and the data 227A, 227B.

[0072] On the displays 219A, 219B, there are the touch panels 230A, 230B as input interfaces.

[0073] Figure 4 is a hardware configuration diagram of the server 3.

[0074] In the server 3, the processor 325, the memory 328, and the network communication machine 320 are connected to each other via a bus 340 connecting the respective constituent elements. The network communication machine 320 is connected to an antenna 323 that transmits and receives network communication signals.

[0075] The memory 328 stores the program 326 and the data 327.

[0076] Figure 5 is a functional block diagram of the image sharing program.

[0077] The image sharing program 400 includes a virtual object display control section 410, a terminal type setting section 420, an individual display object setting section 430, a virtual figure display control section 440, and a communication control section 450.

[0078] An individual display object refers to a virtual object among virtual objects that is set as an individual display object, and is a virtual object that is displayed on a display of another portable information terminal in a presentation manner viewed from a viewpoint of a specific portable information terminal. The specific portable information terminal is a master terminal, and the individual display object is displayed on a slave terminal, i.e., another portable information terminal, in the presentation manner on the master terminal (visual effect).

[0079] A virtual object that is not set as an individual display object is referred to as a general display object. The general display object is displayed on a display of each portable information terminal in a presentation manner viewed from a viewpoint of the portable information terminal itself. Thus, on a display of a master terminal, the general display object is displayed in a presentation manner viewed from a viewpoint of the master terminal, and on a display of a slave terminal, the general display object is displayed in a presentation manner viewed from a viewpoint of the slave terminal.

[0080] The image sharing program 400 is installed in each portable information terminal that constitutes the image sharing system 100. A processor of each portable information terminal loads it into a memory of the portable information terminal and executes, thereby realizing functions of each section.

[0081] The virtual object display control section 410 performs display setting of virtual objects, such as setting a display position, an orientation, a display color, and the like of a virtual object arranged in a virtual space.

[0082] The terminal type setting section 420 sets the present portable information terminal as a master terminal when receiving a setting operation of the master terminal, and generates a notification message that notifies other portable information terminals connected to the image sharing system 100 that the present portable information terminal has been set as the master terminal. In addition, when receiving a notification message that another portable information terminal has been set as the master terminal from the other portable information terminal, the present portable information terminal is set as a slave terminal.

[0083] The individual display object setting section 430 sets at least one of virtual objects as an individual display object when receiving an operation of setting the virtual object as the individual display object. In addition, the setting operation of the individual display object can also be used as an operation of setting the present portable information terminal as the master terminal. In this case, when the individual display object setting section 430 receives the setting operation of the individual display object, the terminal type setting section 420 is notified that the present portable information terminal is set as the master terminal.

[0084] Alternatively, the operation of setting an individual display object from the virtual object can be received after receiving an operation of setting the local device as the host terminal.

[0085] The virtual image display control section 440 controls to display the virtual image corresponding to the user of each portable information terminal on the display of the present portable information terminal, and transmits the display position, the action, the display color, and the like of the virtual image of the present portable information terminal as virtual image display information to the other portable information terminals. In addition, when the virtual image display information is received from the other portable information terminals, the virtual image corresponding to the other user is displayed on the display of the present portable information terminal according to the information. In addition, examples of using the position of the virtual image to indicate the viewpoint sharing state are described in the embodiments described later, and the display control of the display position of the virtual image in this case is also performed by the virtual image display control section 440. In addition, in the case where the other user can see the real space using the see-through type HMD, the virtual image can not be used, and the virtual image in the following embodiments can be replaced with the image of the user who can be seen.

[0086] The communication control section 450 performs communication control of direct communication with the other portable information terminals and network communication via the access points 23A, 23B. The communication control section 450 performs the transmission and reception of all data required for displaying the virtual space, for example, the transmission and reception of the notification message data of the setting as the host terminal, the transmission and reception of the notification message data of the setting of the virtual object as the individual display object, and the like.

[0087] In Figure 6 , Figure 7 FIG. 6 is a diagram showing an example of the display of the virtual space of the present embodiment.

[0088] Figure 6 is a diagram showing an image of the virtual space viewed from a normal terminal viewpoint. Here, the terminal viewpoint refers to a position within the virtual space such that when the display processing system of the portable information terminal, for example, the HMD 1A, 1B, displays the virtual object within the virtual space on the display 119A, 119B of the HMD 1A, 1B, respectively, the virtual object is displayed in a manner of being viewed from the position of the terminal viewpoint.

[0089] Figure 7 is a diagram showing an image of the virtual space viewed from each terminal viewpoint at the time point when the visual sharing of the selected virtual object is started. Here, the visual sharing refers to a method of performing image control on the selected virtual object so that each user sees the same orientation of the virtual object. Hereinafter, the virtual object which is the target of the visual sharing is referred to as an individual display object. In addition, the virtual object which is not the individual display object is referred to as a general display object.

[0090] In the following explanation, it is assumed that the virtual object 8 is designated as an individual display object, and the virtual objects 6, 7 are not designated as individual display objects but are general display objects. Thus, in the following explanation, it is sometimes described as "individual display object (virtual object 8)".

[0091] In addition, the visual sharing of the individual display object is specifically performed by making the presentation of the individual display object viewed from the slave terminal (here, the HMD IB) consistent with the presentation of the individual display object viewed from the master terminal (here, the HMD IA).

[0092] As Figure 7 indicated, in the slave terminal (HMD IB), when the visual sharing is started, the display of the individual display object is switched in conformity with the presentation seen from the master terminal (HMD IA).

[0093] Figure 8 is a diagram showing the state of the virtual space in the visual sharing. After the visual sharing is started, when the master terminal (HMD IA) changes the position and posture of the individual display object (virtual object 8), the display of the individual display object (virtual object 8) of the slave terminal (HMD IB) is changed in conformity therewith. Thus, the presentation of the individual display object, that is, the virtual object 8, becomes the same presentation in the slave terminal (HMD IB) and the master terminal (HMD IA) synchronously. On the other hand, the presentation of the general display objects, that is, the virtual objects 6, 7, is displayed in the presentation viewed from the viewpoint of the slave terminal (HMD IB), so the presentation of the general display objects (virtual objects 6, 7) is different in the master terminal (HMD IA) and the slave terminal (HMD IB).

[0094] In addition, even in the visual sharing, the background of the virtual space in the slave terminal (HMD IB) is displayed on the display 119B using the image viewed from the viewpoint position of the slave terminal (HMD IB). Thus, the display 119B displays an image in which the individual display object 8 configured using the image viewed from the viewpoint direction of the master terminal (HMD IA) is arranged on the background of the virtual space configured using the image viewed from the viewpoint position of the slave terminal (HMD IB), and the general display objects 6, 7 configured using the image viewed from the viewpoint direction of the slave terminal (HMD IB) are arranged.

[0095] Figure 9is a diagram showing an example of the state of the virtual space after the end of the visual sharing. After the end of the visual sharing, the individual display object (virtual object 8) returns to the general display object, but the position and attitude of this general display object at this time are the last position and attitude of the individual display object (virtual object 8) with respect to the host terminal (HMD1A). At this time, in the slave terminal (HMD1B), the general display object disposed in the virtual space is displayed in the state viewed from the terminal viewpoint of the slave terminal (HMD1B) as with other general display objects. In addition to this, as a process at the time of the end of the visual sharing, a method of returning to the state before the start of the visual sharing (the state of Figure 6 ) can also be adopted.

[0096] In addition, the host terminal (HMD1A) and the slave terminal (HMD1B) are not fixed and can appropriately exchange roles.

[0097] <Display processing example of individual display object>

[0098] The display processing example of the individual display object will be described with reference to Figure 10 , Figure 11 . Figure 10 is a diagram showing the definition coordinate system of the individual display object, Figure 11 is a diagram showing the position and orientation of the virtual object definition coordinate system with respect to the virtual space coordinate system.

[0099] First, the data for drawing each virtual object is described using the virtual object definition coordinate system {X T , Y T , Z T} ( Figure 10 ).

[0100] When drawing the virtual object in the virtual space, in the virtual space coordinate system {X, Y, Z} shown in Figure 11 , the position of the virtual object origin O T in the virtual space (hereinafter also referred to as "disposition position R T ") and the orientation of the virtual object definition coordinate system {X T , Y T , Z T} in the virtual space (hereinafter also referred to as "disposition orientation") are decided, whereby drawing can be performed ( Figure 11 ). The "orientation in the virtual space" specifically refers to the orientation of the virtual object definition coordinate system {X T , Y T , Z T} with respect to the virtual space coordinate system {X, Y, Z}.

[0101] Hereinafter, the signs of points in space, directions, rotations in space are expressed by quaternions unless otherwise specified. In addition, quaternions expressing directions are normalized to have a magnitude of 1. Further, the real part of quaternions expressing points, directions is 0. The conjugate quaternion of a quaternion q is denoted by q * .

[0102] Here, the configuration orientation of the virtual object is expressed by a rotation Q T , and the virtual space coordinate system {X, Y, Z} in the virtual space is expressed by N X , N Y , N Z , respectively. When the directions of the respective coordinate axes of the virtual object defining coordinate system {X T , Y T , Z T} in the virtual space are denoted by N TX , N TY , N TZ , respectively, these directions are determined by the rotation Q T as described below.

[0103] N TX = Q T N X Q T * ... (1)

[0104] N TY = Q T N Y Q T * ... (2)

[0105] N TZ = Q T N Z Q T * ... (3)

[0106] Thus, if the configuration position R T and the configuration orientation Q T are determined, the display position and the attitude of the virtual object can be specified.

[0107] Hereinafter, the display state of the virtual object is expressed by the group [R T , Q T ] of the configuration position and the configuration orientation. In addition, the coordinate system of the virtual space can be shared between the master terminal (HMD1A) and the slave terminal (HMD1B), and the information of the position and the attitude of each virtual object is shared through communication between the master terminal (HMD1A) and the slave terminal (HMD1B) or via the server 3.

[0108] Methods for sharing coordinate systems are described, for example, in the international public notice WO2021053805, which is a known technology, so detailed descriptions are omitted here.

[0109] First use Figure 12 This describes the state before visual sharing begins. Figure 12 Is Figure 6 This is a diagram showing the positional relationship between the two terminal viewpoints and the virtual object, viewed from above. Let the display state of virtual object 8 before visual sharing begins be [R]. T0 Q T0 ].

[0110] Figure 13 It means Figure 7 A diagram showing the positional relationship between the two terminal viewpoints and the virtual object at the start of visual sharing. In the slave terminal HMD1B, as shown by the dashed line, the orientation of the virtual object 8, which is displayed individually, changes as visual sharing begins.

[0111] Next, the details of the display processing of virtual objects will be explained. First, Figure 14 This is a diagram showing the baseline display configuration in the host terminal at the start of visual sharing. Figure 15 This is a diagram showing the display baseline configuration in the slave terminal at the start of visual sharing.

[0112] The position and orientation of individual display objects in the display baseline configuration are... Figure 13 The initial state of the configuration position and orientation of individual display objects in each portable information terminal, namely the host terminal (HMD1A) and the slave terminal (HMD1B), at the start of visual sharing.

[0113] like Figure 14 As shown, the individual display objects seen from the host terminal (HMD1A) are located at their initial configuration positions R. T0 Similarly, as Figure 15 As shown, the individual display objects seen from the slave terminal (HMD1B) are located at their initial configuration positions R. T0 .

[0114] Figure 14 , Figure 15 In this context, the initial positions of the host terminal (HMD1A) and slave terminal (HMD1B) at the start of visual sharing are referred to as the terminal reference points in each terminal.

[0115] At the start of visual sharing, a horizontal reference direction is set for each individual display object (virtual object 8). The initial configuration position R containing the individual display object is then set from the terminal reference point. T0defined as a projection point. The direction from the initial configuration position R T0 of the individual display object to the projection point is decided as the horizontal reference direction.

[0116] Figure 14 The horizontal reference direction of the host terminal (HMD1A) is denoted as D (M) H0 , Figure 15 The horizontal reference direction of the slave terminal (HMD1B) is denoted as D (S) H0 .

[0117] Next, the processing of reflecting the change in the configuration position and the configuration attitude of the individual display object by the host terminal (HMD1A) to the change in the position and the attitude of the individual display object displayed by the slave terminal (HMD1B) in the visual sharing will be described.

[0118] First, in order to describe the introduction of several symbols. Let the current configuration position of the individual display object (virtual object 8) in the visual sharing be R (#) T , and the configuration orientation be Q (#) T . The superscript ( ) indicates the difference between the master and the slave, (M) indicates the host, and (S) indicates the slave. In order to transform the configuration position R (M) T and the configuration orientation Q (M) T changed by the host terminal (HMD1A) into the configuration position R (S) T and the configuration orientation Q (S) T of the slave terminal (HMD1B) used, the configuration position change amount and the rotation of the configuration orientation are performed in accordance with the difference from the horizontal reference direction.

[0119] Figure 16 is a diagram showing the relationship between the horizontal reference direction of the host terminal and the horizontal reference direction of the slave terminal.

[0120] By performing a transformation of rotating the horizontal reference direction D (M) H0 around the vertical axis to coincide with the horizontal reference direction D (S) H0 , the configuration position R (M) T and the configuration orientation Q (M) TThe transformation is performed. Thus, display of the individual display object (virtual object 8) in the slave terminal (HMD1B) can be performed as viewed from the direction in which the host terminal (HMD1A) is located. The rotation representing this transformation is referred to as MS rotation transformation, and when Q (MS) V is used, the rotation transformation parameter is represented by Q (MS) V This can be written as follows.

[0121] Q (MS) V = cos(θ / 2) + sin(θ / 2)N V ... (4)

[0122] Here, N V is a unit vector of the vertical upward direction expressed by a quaternion. θ is the angle from the horizontal reference direction D (M) H0 of the host terminal in the display reference configuration to the horizontal reference direction D (S) H0 of the slave terminal (HMD1B) calculated clockwise when facing the vertical upward direction.

[0123] Further, more specific writing of equation (4) can be expressed as follows.

[0124] Q (MS) V = cos(θ / 2) + sin(θ / 2)(N V ) X i + sin(θ / 2)(N V ) Y j + sin(θ / 2)(N V ) Z k... (4')

[0125] Here, i, j, k are units other than 1 of the quaternion. In the case of complex numbers, only the i term is present.

[0126] From this MS rotation transformation parameter Q (MS) V , the display state of the individual display object (virtual object 8) that should be displayed in the slave terminal (HMD1B) is found as follows.

[0127] R (S) T = R T0 + Q (MS) V (R (M) T - R T0 )Q (MS) V* ... (5)

[0128] Q (S) T = Q (MS) V Q (M) T ... (6)

[0129] According to the above, as long as the reference parameters R T0 , D (M) H0 and D (S) H0 are acquired, and the display state of the individual display object (virtual object 8) in the host terminal (HMD1A) at the time of display, the slave terminal (HMD1B) can display the individual display object (virtual object 8) in the same state as the state viewed from the host terminal (HMD1A) in the orientation in the horizontal plane.

[0130] Here, the initial setting information that the slave terminal (HMD1B) should acquire is the terminal reference point of the host terminal (HMD1A) and the initial arrangement position information (R T0 ) of the individual display object (virtual object 8). The slave terminal (HMD1B) grasps the terminal reference point of its own terminal. In addition, in the case where the slave terminal (HMD1B) starts visual sharing at the same time as the host terminal (HMD1A), R T0 has already been grasped, so the information of R T0 is not needed. D T0 can be found from the terminal reference point of the host terminal (HMD1A) and R (M) H0 , and D T0 (S) H0 can be found from the terminal reference point of the slave terminal (HMD1B) and R T0 . Thus, all the information of the above reference parameters R (M) , D H0 and D (S) H0 is obtained. In the case where the virtual image of the host terminal (HMD1A) is displayed, the head position of the virtual image can be made the terminal reference position. In addition, in the case where the terminal position has been changed from the terminal reference point, the virtual object is displayed in such a manner as to reproduce the position and posture in the virtual space found above on the basis of taking the change into account.

[0131] As for the change in the display state of the individual display object (virtual object 8), the virtual object can also be enlarged or reduced.

[0132] Furthermore, even when entering visual sharing mode, the virtual objects in the host terminal are displayed in the same way as viewed from the viewpoint of the host terminal, so there is no change compared to the way of viewing ordinary display objects.

[0133] In the above description, individual display objects are displayed on the slave terminal as if viewed from the host terminal's perspective. However, the display mode can also be changed to a presentation mode viewed from the position of the slave terminal itself. That is, it becomes such that although the host terminal holds the authority to change the position and orientation of the object, the slave terminal can display it as a normal display object without applying MS rotation transformation. In this case, equations (5) and (6) become the following equation.

[0134] R (S) T =R (M) T ... (7)

[0135] Q (S) T =Q (M) T ... (8)

[0136] The display based on the host terminal viewpoint (Equations (5) and (6)) is called MS transformation display, and the display based on the slave terminal viewpoint (Equations (7) and (8)) is called normal display.

[0137] Furthermore, the method of viewing from the perspective of other terminals can also be applied to terminals other than the host terminal. Based on the terminal reference point of the other terminal and the configuration position of the target object, the rotation transformation corresponding to the MS rotation transformation can be calculated, thus enabling the same display mode transformation. Figure 13 In other words, it refers to the horizontal reference direction D. (M) H0 Replace it with the horizontal reference orientation of other terminals that need visual sharing. This is useful when temporarily confirming how it is presented on other terminals, etc.

[0138] When visual sharing begins, the host terminal is first determined. If a user is the primary presenter in a meeting or similar setting, they are considered the designated host terminal for that setting. Alternatively, a portable information terminal that wishes to become the host terminal can declare itself as such. Other slave terminals, wanting to engage in visual sharing with the host terminal, specify a general display object as the target to initiate the sharing process.

[0139] The virtual object can be specified by any method, such as pointing with a beam of light from the controller, indicating with a virtual object representing the user's hand, looking at it with eyes, or selecting a number to assign to each virtual object.

[0140] Figure 17 is a view showing an example of host terminal indication.

[0141] As shown in Figure 17 in order to indicate whose terminal is the host terminal, in a case where a virtual image AVT_A corresponding to the user A of the host terminal and a virtual image AVT_B corresponding to the user B of the slave terminal are displayed, a graphical indication showing the user of the host terminal can be attached to the virtual image corresponding to the user of the host terminal. As an example of the graphical indication, for example, the host terminal mark MRK_M or the host terminal indication MSG_MA including a text indication can be attached to the virtual image AVT_A.

[0142] Figure 18 is a view showing another example of host terminal indication.

[0143] The host terminal can be set for each specific virtual object. That is, in a case where a plurality of virtual objects exist in the virtual space, the host terminal of each virtual object can be different. Figure 18 In the view, the host terminal of the virtual object 8 is HMD1A, so the host terminal indication MSG_MA "the host terminal is A" is attached to the virtual object 8. In addition, the slave terminal of the virtual object 8 is HMD1B.

[0144] In addition, Figure 18 In the view, the host terminal of the virtual object 7 is HMD1B, so the host terminal indication MSG_MB "the host terminal is B" is attached to the virtual object 7. In addition, the slave terminal of the virtual object 7 is HMD1A.

[0145] In addition, Figure 18 In the view, the virtual object 6 is a general display object.

[0146] The visual sharing of the individual display object can also be performed among three or more portable information terminals. At this time, one portable information terminal is the host terminal and the other portable information terminals are the slave terminals.

[0147] In addition, the slave terminal can be temporarily given the authority of the host terminal so as to be able to change the position and posture of the individual display object. In a case where the host terminal becomes the slave terminal midway, the terminal reference point information of the portable information terminal newly becoming the host terminal is acquired, and a rotation transformation for performing the posture information transformation is calculated.

[0148] Further, other portable information terminals can subsequently join the group in which visual sharing is being performed as slave terminals. Upon joining, as long as initial setting information is acquired, the display state that should be displayed in the own portable information terminal can be found from the display state of the individual display object in the host terminal thereafter. The initial setting information can be received directly from the host terminal, or can be received from other slave terminals or the server 3.

[0149] In a case where the host terminal is decided in advance in a conference or the like, the host terminal alone starts visual sharing, and initial setting information is set. On this basis, the slave terminals that are to perform visual sharing appropriately acquire the initial setting information and start visual sharing.

[0150] Further, the virtual objects in which visual sharing is performed can be plural, and the host terminal can differ for each virtual object.

[0151] Further, display control of the virtual objects can be performed by the server 3 entirely. In this case, each portable information terminal transmits initial setting information of each portable information terminal to the server 3. The initial setting information in this case is information of a terminal reference position of each portable information terminal, and an initial arrangement position of the individual display object when becoming the host terminal. Further, the portable information terminal that becomes the host terminal transmits change information of a change made by the host terminal to the display state of the individual display object to the server 3. The server 3 performs display control of the individual display object of each portable information terminal.

[0152] Figure 19 is a flowchart showing the processing flow of the image sharing system of the present embodiment, and shows a manner in which display control of the individual display object is performed in the portable information terminal. Hereinafter, a case in which one of three virtual objects is designated as the individual display object and displayed between user A and user B as shown in Figure 1 is explained.

[0153] First, when the image sharing processing, in detail, the visual sharing processing of the individual display object is started between HMD1A and HMD1B, at least one or more virtual objects are displayed in HMD1A and HMD1B respectively, and setting of a display reference arrangement is performed (S01A, S01B).

[0154] Next, HMD1A receives processing in which it is set as the host terminal. In addition, in this step, HMD1A receives designation of the individual display object (S02A). Along with this processing, HMD1A notifies HMD1B that it has become the host terminal. Upon receiving this notification, HMD1B performs slave terminal setting (S02B).

[0155] The host terminal (HMD1A) transmits initial setting information to the slave terminal (HMD1B) (S03), and the slave terminal (HMD1B) receives the information (S04). The initial setting information includes terminal reference point information of the host terminal (HMD1A). In addition, the initial setting information includes information indicating which of the virtual objects is the individual display object and information indicating an initial arrangement position thereof.

[0156] The slave terminal (HMD1B) calculates MS rotation transformation parameters Q for performing display state transformation using the host terminal (HMD1A) reference point information in the received initial setting information and terminal reference point information of the own portable information terminal (MS) V (S05).

[0157] Next, a display mode is set in the HMD1B (S06). Here, the display mode includes host terminal viewpoint display using MS rotation transformation (equations (5), (6)) and normal display (equations (7), (8)).

[0158] The HMD1A and the HMD1B each measure the position and the posture of the respective terminals (S07A, S07B).

[0159] Then, when the host terminal (HMD1A) changes the display state of the individual display object, that is, changes either or both of the position and the posture of the individual display object (S08), the host terminal (HMD1A) transmits the change information to the slave terminal (HMD1B) (S09), and the slave terminal (HMD1B) receives the information (S10).

[0160] In the slave terminal (HMD1B), the display of the individual display object is updated in accordance with the change in the position and the posture of the terminal of the host terminal (HMD1A) and the display state of the individual display object (S11). Thus, on the display 119B of the HMD1B, the virtual object 8 viewed from the viewpoint of the HMD1A can be displayed, so the user A and the user B can view the same virtual object from the same viewpoint. In other words, the user A and the user B can share the viewpoint.

[0161] Until the user indicates to end the visual sharing (S12A: "No", S12B: "No"), the process returns to steps S06A, S06B to continue the display of the individual display object, and if there is an indication to end the visual sharing (S12A: "Yes", S12B: "Yes"), an ending process is performed. In the ending process, the terminal settings are released on the respective terminals, the display of the individual display object is returned to the display of the general display object, and the visual sharing is ended (S13A, S13B).

[0162] The method of restoring the display of the individual display object (virtual object 8) to the display of the general display object can be a method of restoring to the display state before the start of the visual sharing or a method of restoring to the display state in the host terminal (HMD1A) at the end of the visual sharing. As to which method to restore, it can be set to a prescribed value of the visual sharing method or it can be switched by receiving the selection of the user A of the host terminal (HMD1A) at the end of the visual sharing.

[0163] Figure 20A 、 Figure 20B is a flowchart showing the processing flow of the image sharing system of the present embodiment, showing the manner of performing the display control of the individual display object in the server. In this example, the display control is performed by the server receiving the information for the transformation, but the flow is basically the same as that of Figure 19 , so the same step numbers are used for the steps common to Figure 19 .

[0164] First, when the image sharing processing, in detail, the visual sharing processing of the individual display object is started between HMD1A and HMD1B, at least one or more virtual objects are displayed in HMD1A and HMD1B respectively, and the setting of the display reference configuration is performed (S01A, S01B).

[0165] Next, HMD1A receives the processing of setting itself as the host terminal. In addition, in this step, HMD1A receives the designation of the individual display object. At the same time, the host terminal (HMD1A) notifies the server 3 and HMD1B that the own terminal has become the host terminal, and the server 3 and HMD1B receive the notification (S02A, S02S, S02B).

[0166] The host terminal (HMD1A) and the slave terminal (HMD1B) respectively transmit the initial setting information to the server (S03A, S30B), and the server 3 receives the information (S04).

[0167] The server 3 calculates the rotation transformation parameter Q (MS) V (S05) using the host terminal (HMD1A) reference point information and the slave terminal (HMD1B) terminal reference point information in the received initial setting information.

[0168] The slave terminal (HMD1B) sets the display mode to the own terminal viewpoint or the host terminal viewpoint and transmits it to the server 3 (S30). The server 3 receives the display mode setting information (S31).

[0169] HMD1A and HMD1B respectively measure the position and posture of each terminal and transmit it to the server 3 (S22A, S22B), and the server 3 receives the information (S22S).

[0170] Then, when the host terminal (HMD1A) changes either or both of the position and the posture of the individual display object (S08), the change information is transmitted from the HMD1A to the server 3 (S09), and the server 3 receives the information (S23). The server 3 calculates the rendering information of the individual object (S24) in order to use for each terminal, based on the changes in the terminal positions and postures of the host terminal (HMD1A) and the slave terminal (1B) and the positions and postures of the individual display objects, and transmits the rendering information to the host terminal (HMD1A) and the slave terminal (HMD1B).

[0171] The host terminal (HMD1A) and the slave terminal (HMD1B) update the display of the individual display objects using the rendering information received from the server 3 (S11A, S11B). Thereby, on the display 119B of the HMD1B, the virtual object 8 viewed from the viewpoint of the HMD1A can be displayed, so the user A and the user B can view the same virtual object from the same viewpoint. In other words, the user A and the user B can share the viewpoint.

[0172] Until the user indicates to end the visual sharing (S12A: "No", S12B: "No", S25: "No"), the image sharing processing is continued, and if there is an indication to end the visual sharing (S12A: "Yes", S12B: "Yes", S25: "Yes"), the terminal settings of each terminal are released, the display of the individual display objects is returned to the display of the general display objects, and the visual sharing is ended (S13A, S13B, S26).

[0173] According to the present embodiment, the virtual object viewed from the viewpoint of the host terminal can be viewed from the slave terminal, so even if the user of the host terminal and the user of the slave terminal are located at different positions and orientations, the presentation of the same virtual object can be shared. Thereby, it is expected that in, for example, work assistance and surgery assistance, learning assistance, education assistance, and the like, the presentation of the virtual object can be shared between different users, understanding can be improved, and communication can be smoothly performed.

[0174] <Modification Example 1>

[0175] The authority to change the display state of the individual display object (virtual object 8) (hereinafter referred to as the individual display change authority) can be transferred between terminals, rather than being fixed to the host terminal (HMD1A). In the case where a plurality of users take turns to explain using the individual display object (virtual object 8), if the terminal of the user who is explaining has the authority, it is expected that an effect that the explanation is easily performed can be achieved.

[0176] <Modification Example 2>

[0177] As for the individual display change authority, the authority can be automatically transferred to the terminal that has performed the display state change of the individual object without issuing an instruction to transfer the authority. When there are terminals that are to perform the display change almost simultaneously, the authority is transferred to the terminal that has performed the display change first. In order to avoid confusion due to operation conflicts, the individual display change authority can be retained in the terminal for a certain time after the authority is transferred and after the operation of the terminal is completed.

[0178] If a terminal that has accepted the authority transfer of the host terminal (HMD1A) is denoted by "M1" and other terminals are denoted by "S", and the rotation angle from the direction of "M1" to the direction of "S" with the initial configuration position as the center point is denoted by θ1, the rotation transformation parameter Q is defined by the following equation (M1S) V .

[0179] Q (M1S) V = cos(θ1 / 2) + sin(θ1 / 2)N V ... (9)

[0180] Here, "S" is a notation for representing the slave terminal (HMD1B) including the host terminal (HMD1A) that temporarily becomes the slave terminal (HMD1B), and it is noted that the value of θ1 is different for each terminal. By using the rotation transformation parameter Q (M1S) V , the display state [R (M1) T , Q (M1) T ] of the individual display object (virtual object 8) in the temporary host terminal (HMD1A) can be transformed into the display state in the other slave terminal (HMD1B) as in the basic form.

[0181] R (S) T = R T0 +Q (M1S) V (R (M1) T -R T0 )Q (M1S) V * ... (10)

[0182] Q (S) T =Q (M1S) V Q (M1) T ... (11)

[0183] <Modified Example 3>

[0184] The initial arrangement position R of the individual display object (virtual object 8) in the slave terminal (HMD1B) can be changed to a position R specific to each slave terminal (HMD1B) T0 (S) T0 In addition, the presentation manner viewed from the host terminal (HMD1A) in the slave terminal (HMD1B) can also be adjusted individually by the amount of rotation Q (ΔS) T Figure 21 is a display example of Modification 3 of the first embodiment.

[0185] R (S) T = R (S) T0 + Q (MS) V (R (M) T - R T0 ) Q (MS) V * ... (12)

[0186] Q (S) T = Q (ΔS) T Q (MS) V Q (M) T ... (13)

[0187] According to Modification 3, in the case where the slave terminal does not have the authority to change the position and orientation of the individual display object (virtual object 8), the individual display object (virtual object 8) can be temporarily viewed from the direction desired by the user.

[0188] In addition, in the case where the arrangement reference point at the start of individual display is a position where the individual object is not easily viewed, etc., by moving it to a position where the terminal user can easily view it, the convenience can be improved. In addition, this method can also be applied to general display objects. The virtual object can be temporarily viewed from the direction desired by the user without affecting the presentation manner of the virtual object to other users. At this time, for the individual display object (virtual object 8) or the general display object in the state where no change has been implemented in the own portable information terminal, they can be displayed by changing their display manner, for example, they can be completely cleared from display, or they can be displayed in a faded (semi-transparent) manner, etc. Alternatively, the virtual object, etc. in the display state where a change has been implemented in the own portable information terminal can also be displayed by changing the display manner, for example, they can be displayed in a faded manner.

[0189] <Modification 4> ​​

[0190] As for a two-dimensional object such as a comment, its viewing direction is only the direction in which the comment or the like is easily seen, so only the content is shared, and the orientation change of the object is not required to be shared. It is an individual display object (virtual object 8) although it is not linked with the orientation change operation of other portable information terminals, so it is called an independent display object 5. For example, for an object having a whiteboard function, the content written on the object is shared between the terminals of visual sharing in a position and orientation easily seen from the viewpoint of each terminal. Figure 22 Fig. 4 is a display example of a modification 4 of the first embodiment.

[0191] [Second Embodiment] [Second Embodiment]

[0192] In the first embodiment, the method of individual display is used to realize the same viewing of the virtual object of interest at each terminal. In this second embodiment, the method of making each terminal view the virtual space from the reference display direction set for the entire virtual space, that is, by making the terminal viewpoint position the same position, is used to realize visual sharing. In this case, when the virtual image of each user is displayed, the display of the virtual images is overlapped. Several methods can be considered for the display method of the virtual image. Figure 23 Fig. 5 is a display example of the second embodiment.

[0193] Figure 23 Only the host virtual image of the virtual images at the same viewpoint position is displayed. Thus, the virtual image can be displayed without a particularly unnatural feeling on the display.

[0194] The host virtual image is, for example, the virtual image of a central person such as the center of the discussion at that time or an explainer. The terminal corresponding to the host virtual image is the host terminal (HMD1A). The slave terminal (HMD1B) other than the host terminal (HMD1A) uses the terminal viewpoint of the host terminal (HMD1A) as the shared viewpoint.

[0195] When the shared viewpoint is used, the own virtual image (part of the hand, etc.) seen by the slave terminal (HMD1B) can be that of the host virtual image, or the own virtual image can be displayed on the basis of adjusting the height. If the own body or a substitute thereof can be seen, a feeling of floating in the air is not felt, and a sense of security can be obtained. Alternatively, in a case where the motion of the host virtual image is different from the motion of the own motion and a feeling of unnaturalness is felt, the virtual image can be made invisible.

[0196] As for the slave virtual image at the original position corresponding to the slave terminal (HMD1B), display can be stopped during visual sharing. However, because the original place is to be returned after visual sharing, display can be performed at the original place in a form in which it can be understood that visual sharing is being performed, for example, as a semi-transparent pale object.

[0197] In addition, the authority to change the position and orientation of the virtual object can be limited to the host terminal (HMD1A).

[0198] In order to indicate that the visual sharing position is entered, the voice of the slave virtual image that is performing the visual sharing can be emitted from the visual sharing position. Alternatively, in order to easily understand who is speaking, it can also be emitted from the original virtual image position.

[0199] In the case of entering the visual sharing position, in order to understand the change in position, the virtual image can be caused to walk to the position. Alternatively, in order to save time, it can also be instantaneously moved to the visual sharing position.

[0200] According to the second embodiment, as with the first embodiment, the viewpoint of the host terminal and the viewpoint of the slave terminal can be shared. Furthermore, it is expected that the effect that the user of the slave terminal easily grasps the positional relationship between the viewpoint of the host terminal and the virtual object will be achieved.

[0201] <Modification Example 1>

[0202] Figure 24 is a display example of Modification Example 1 of the second embodiment.

[0203] Figure 24 In the second embodiment, because the height of the viewpoint of each terminal is not fixed, when the visual sharing is performed by the slave terminal (HMD1B) whose viewpoint height is different from that of the host terminal (HMD1A), it can feel unnatural if the viewpoint of the host terminal (HMD1A) is used. Therefore, a plumb line that passes through the position of the viewpoint of the host terminal (HMD1A) is defined as a shared viewpoint line. Then, when the viewpoint sharing is performed by the slave terminal (HMD1B), the position of the terminal viewpoint height of the own portable information terminal on the shared viewpoint line is used as the viewpoint at the time of the visual sharing. Thus, it does not feel unnatural as described above.

[0204] In addition, when the own terminal is the slave virtual image that is performing the visual sharing, the own virtual image can be made visible or invisible.

[0205] According to Modification Example 1, as with the effect of the above-described second embodiment, the viewpoint of the host terminal and the viewpoint of the slave terminal can be shared. Furthermore, it is expected that the effect that the user of the slave terminal easily grasps the positional relationship between the viewpoint of the host terminal and the virtual object will be achieved.

[0206] <Modification Example 2>

[0207] Figure 25 is a display example of Modification Example 2 of the second embodiment.

[0208] As Figure 25As shown, in Modification 2, the host virtual image can be replaced. In a case where, while HMD 1A is the host terminal, user A is explaining with respect to an object, user B of HMD 1B speaks in the middle, HMD 1B becomes the new host terminal, and the host virtual image corresponding to user B of HMD 1B can be displayed at the same position. In this case, visual sharing can be performed at the viewpoint position of the host virtual image when HMD 1B is the slave terminal, or the individual display object (virtual object 8) targeted for visual sharing can be displayed only at the original position of the own portable information terminal as in the first embodiment. HMD 1B can remain at the position of the host virtual image after the speech ends, or can return to the original position.

[0209] According to Modification 2, as with the effects of the above-described second embodiment, the viewpoint of the host terminal and the viewpoint of the slave terminal can be shared. Furthermore, it is expected that the user of the slave terminal can easily intuitively grasp that he or she is sharing the viewpoint of the host terminal.

[0210] <Modification 3>

[0211] Figure 26 Fig. 9 is a diagram showing a display example of Modification 3 of the second embodiment.

[0212] As shown in Fig. 9, in Modification 3, the slave virtual image that is performing visual sharing can be displayed side by side behind the host virtual image. Figure 26

[0213] According to Modification 3, as with the effects of the above-described second embodiment, the viewpoint of the host terminal and the viewpoint of the slave terminal can be shared. Furthermore, it is expected that the users of the host terminal and the slave terminal can easily intuitively grasp who has entered the shared viewpoint position.

[0214] <Modification 4>

[0215] Figure 27 Fig. 10 is a diagram showing a display example of Modification 4 of the second embodiment.

[0216] As shown in Fig. 10, in Modification 4, the virtual image that is performing visual sharing can be displayed in a reduced size at the visual sharing position. Figure 27

[0217] According to Modification 4, as with the effects of the above-described second embodiment, the viewpoint of the host terminal and the viewpoint of the slave terminal can be shared. Furthermore, because the display area of the virtual image can be reduced, the display area of the virtual object can be easily ensured.

[0218] Alternatively, the host virtual image can be made to remain at the original size or a smaller reduction rate than the slave virtual image, so as to be conspicuous.

[0219] <Modification 5> ​​

[0220] Figure 28 Fig. 6 is a diagram showing a display example of a modification 5 of the second embodiment.

[0221] Figure 28 In the modification 5 shown, a shared avatar can be used in a plurality of portable information terminals. The portable information terminals using the shared avatar AVT_SH can perform visual sharing. The body control of the avatar is limited to a certain portable information terminal. Since there is a possibility of confusion when allowing sound emission from a larger number of portable information terminals, the sound emission can also be limited to a certain number of terminals.

[0222] In order to distinguish from the avatar used alone, the appearance of the shared avatar AVT_SH can be made different from the appearance of the other avatar used alone.

[0223] <Modification 6>

[0224] Figure 29 Fig. 7 is a diagram showing a display example of a modification 6 of the second embodiment.

[0225] Figure 29 In the modification 6 shown, the body freedom degree can be increased differently from the appearance of a normal human being, for example, the number of arms is increased, and the like, and is commonly controlled by a plurality of terminals.

[0226] Thereby, a complicated operation can be performed on the object.

[0227] In addition, it is also possible to perform work assistance from the same body position from the viewpoint of teaching.

[0228] Figure 29 The case where the number of arms is increased is shown in Fig. 8. The body freedom degree of the avatar is divided, and each portable information terminal is given control right for each freedom degree.

[0229] For example, a normal body part is used as a first freedom degree, and an additional lower arm part is used as a second freedom degree. In addition, in the case where work is performed with the lower arm, there is a possibility that the terminal user controlling the second freedom degree cannot perform control as intended when the trunk part of the body moves, so in this case, the movement of the trunk part can be made to be below a certain speed including stillness, or the like, and a restriction is imposed on the control of the first freedom degree.

[0230] Alternatively, the lower arm can be completely separated from the movement of the first freedom degree.

[0231] Furthermore, it is also possible to separate from the movement of the first freedom degree for a part of the time when work is performed. In the display mode, the target body part of the first freedom degree (for example, the upper arm) and the target body part of the second freedom degree (for example, the lower arm) can exist in a portion overlapping in the virtual space.

[0232] According to the present embodiment, a plurality of portable information terminals are communicatively connected, and the user can also view a virtual object in a presentation mode in which the virtual object is viewed from the viewpoint of another person, so by accepting a user selection, the display mode of the virtual object is switched from an image viewed from the viewpoint position of the user's own portable information terminal to an image viewed from the viewpoint direction of another portable information terminal, and visual sharing can be experienced with the user of the other portable information terminal.

[0233] The above describes an embodiment of the present application, but the structure of the technology implementing the present application is not limited to the above-described embodiment, and various modifications can be considered. For example, the above-described embodiment is described in detail in order to easily understand the present application, and is not limited to necessarily having all of the structures described. Also, a part of the structure of one embodiment can be replaced with the structure of another embodiment, and the structure of one embodiment can be added with the structure of another embodiment. All of these belong to the scope of the present application. Also, the values and messages and the like appearing in the text and the figures are only examples, and the effects of the present application are not impaired even if different ones are used.

[0234] Also, the programs described in each of the processing examples can be separate programs, or can be constituted by a plurality of programs as one application. Also, the order of performing each processing can be changed to execute the programs.

[0235] The functions and the like of the above-described present application can be implemented in part or all of them by hardware such as an integrated circuit, for example. Also, it can be implemented by software by a microprocessor unit, a CPU and the like interpreting and executing an action program implementing each function and the like. Also, the implementation range of the software is not limited, and hardware and software can be used together. Also, a part or all of each function can be implemented by a server. Also, the server can be any as long as it can perform a function in cooperation with other components via communication, and is, for example, a local server, a cloud server, an edge server, a network service and the like. The programs, tables, files and the like information implementing each function can be stored in a storage device such as a memory, a hard disk, an SSD (Solid State Drive) or the like, or an IC card, an SD card, a DVD and the like recording medium, or in a device on a communication network.

[0236] Also, the control lines and the information lines shown in the figures show only the necessary parts for explanation, and do not necessarily show all of the control lines and the information lines on the product. In fact, it can be considered that almost all of the structures are connected to each other.

[0237] The above-described embodiment includes the following modes.

[0238] (Postscript 1)

[0239] An image sharing system,

[0240] including a first portable information terminal and a second portable information terminal,

[0241] the first portable information terminal includes:

[0242] a first display;

[0243] a first communication machine that performs data transmission and reception with the second portable information terminal; and

[0244] a first processor that controls the first display and the first communication machine, wherein

[0245] the first processor executes visual sharing control in which a display mode of an individual display object constituted by a virtual object selected as a visual sharing target is switched from an image viewed from a viewpoint position of the first portable information terminal to an image viewed from a viewpoint direction of the second portable information terminal based on the data received via the first communication machine, in accordance with a selection operation of a user of the first portable information terminal or a user of the second portable information terminal,

[0246] the first display displays the individual display object using the image viewed from the viewpoint direction of the second portable information terminal.

[0247] (Addendum 2)

[0248] An image sharing method that executes the following steps:

[0249] a first step in which a portable information terminal executes visual sharing control in which a display mode of a virtual object selected as a visual sharing target among a plurality of users is switched from an image viewed from a viewpoint position of the portable information terminal to an image viewed from a viewpoint direction of another portable information terminal; and

[0250] a second step in which the portable information terminal displays the virtual object using the image viewed from the viewpoint direction of the other portable information terminal.

[0251] Explanation of Reference Numerals

[0252] 1A: HMD

[0253] 1B: HMD

[0254] 1C: Smartphone

[0255] 1D: Smartphone

[0256] 3: Server

[0257] 5: Individual Display Object

[0258] 6: Virtual Objects

[0259] 7: Virtual Objects

[0260] 8: Virtual Objects

[0261] 9: Network

[0262] 23A: Access Point

[0263] 23B: Access Point

[0264] 100: Image Sharing System

[0265] 111A: External camera

[0266] 111B: External camera

[0267] 113A: Distance sensor

[0268] 113B: Distance sensor

[0269] 114A: RTC

[0270] 114B: RTC

[0271] 115A: Accelerometer

[0272] 115B: Accelerometer

[0273] 116A: Gyroscope sensor

[0274] 116B: Gyroscope sensor

[0275] 117A: Geomagnetic sensor

[0276] 117B: Geomagnetic sensor

[0277] 118A: Positioning sensor

[0278] 118B: Positioning sensor

[0279] 119A: Monitor

[0280] 119B: Monitor

[0281] 120A: Network communication device

[0282] 120B: Network communication device

[0283] 121A: Microphone

[0284] 121B: Microphone

[0285] 122A: Speaker

[0286] 122B: Loudspeaker

[0287] 123A: antenna

[0288] 123B: antenna

[0289] 125A: processor

[0290] 125B: processor

[0291] 126A: program

[0292] 126B: program

[0293] 127A: data

[0294] 127B: data

[0295] 128A: memory

[0296] 128B: memory

[0297] 129A: wireless communication device

[0298] 129B: wireless communication device

[0299] 135A: position and attitude sensor

[0300] 135B: position and attitude sensor

[0301] 136A: position and attitude camera

[0302] 136B: position and attitude camera

[0303] 137A: position and attitude ranging sensor

[0304] 137B: position and attitude ranging sensor

[0305] 140A: bus

[0306] 140B: bus

[0307] 211A: rear camera

[0308] 211B: rear camera

[0309] 212A: front camera

[0310] 212B: front camera

[0311] 213A: ranging sensor

[0312] 213B: ranging sensor

[0313] 214A: RTC

[0314] 214B: RTC

[0315] 215A: acceleration sensor

[0316] 215B: acceleration sensor

[0317] 216A: gyro sensor

[0318] 216B: gyro sensor

[0319] 217A: geomagnetic sensor

[0320] 217B: geomagnetic sensor

[0321] 218A: positioning sensor

[0322] 218B: positioning sensor

[0323] 219A: display

[0324] 219B: display

[0325] 220A: network communication device

[0326] 220B: network communication device

[0327] 221A: microphone

[0328] 221B: microphone

[0329] 222A: speaker

[0330] 222B: speaker

[0331] 223A: antenna

[0332] 223B: antenna

[0333] 225A: processor

[0334] 225B: processor

[0335] 226A: program

[0336] 226B: program

[0337] 227A: data

[0338] 227B: data

[0339] 228A: memory

[0340] 228B: memory

[0341] 229A: wireless communication device

[0342] 229B: wireless communication device

[0343] 230A: touch panel

[0344] 230B: touch panel

[0345] 231A: telephone network communicator

[0346] 231B: telephone network communicator

[0347] 235A: position and posture sensor

[0348] 235B: position and posture sensor

[0349] 236A: position and posture camera

[0350] 236B: position and posture camera

[0351] 237A: position and posture distance measuring sensor

[0352] 237B: position and posture distance measuring sensor

[0353] 240A: bus

[0354] 240B: bus

[0355] 320: network communicator

[0356] 323: antenna

[0357] 325: processor

[0358] 326: program

[0359] 327: data

[0360] 328: memory

[0361] 340: bus

[0362] 400: image sharing program

[0363] 410: virtual object display control section

[0364] 420: terminal type setting section

[0365] 430: individual display object setting section

[0366] 440: virtual image display control section

[0367] 450: communication control section

[0368] AVT_A: virtual image

[0369] AVT_B: virtual image

[0370] AVT_SH: shared virtual image

[0371] MRK_M: master terminal marker

[0372] MSG MA: host terminal designation

[0373] MSG MB: host terminal designation

Claims

1. An image sharing system, characterized in that: It includes a first portable information terminal and a second portable information terminal. The first portable information terminal includes: First display; A first communication device for transmitting and receiving data with the second portable information terminal; and A first processor that controls the first display and the first communication device, wherein... The first processor executes visual sharing control according to the selection operation of the user of the first portable information terminal or the user of the second portable information terminal. In the visual sharing control, the display mode of individual display objects composed of virtual objects that are visual sharing targets is switched from the image viewed from the viewpoint of the first portable information terminal to the image viewed from the viewpoint of the second portable information terminal based on the data received via the first communication device. The first display uses an image viewed from the perspective of the second portable information terminal to display the individual display objects.

2. The image sharing system as described in claim 1, characterized in that: The first processor, based on the difference between the viewing direction of the first portable information terminal and the viewing direction of the second portable information terminal as seen from the individual display object, transforms the position and orientation of the individual display object in the virtual space from the viewing position of the first portable information terminal to the position and orientation of the individual display object in the virtual space from the viewing position of the second portable information terminal. Visual sharing control is performed by switching the image of the individual display object viewed from the first portable information terminal to the image of the individual display object viewed from the second portable information terminal.

3. The image sharing system as described in claim 1, characterized in that: The first processor causes the terminal viewpoint position of the first portable information terminal in the virtual space to overlap with the terminal viewpoint position of the second portable information terminal, thereby switching the image of the individual display object viewed from the first portable information terminal to the image of the individual display object viewed from the second portable information terminal, to perform visual sharing control.

4. The image sharing system as described in claim 1, characterized in that: The first processor performs display control on a general display object consisting of virtual objects that are not visually shared targets, by displaying the image viewed from the viewpoint of the first portable information terminal on the first display screen. The first display simultaneously displays the individual display objects composed of images viewed from the viewpoint of the second portable information terminal, and the general display objects composed of images viewed from the viewpoint of the first portable information terminal.

5. The image sharing system as described in claim 1, characterized in that: The first processor performs display control to display an image on the first display, wherein the image is obtained by configuring individual display objects composed of images viewed from the viewpoint of the second portable information terminal on a virtual space background composed of images viewed from the viewpoint of the first portable information terminal.

6. The image sharing system as described in claim 1, characterized in that: Multiple virtual objects are configured in the virtual space. For a given virtual object, an image viewed from the line of sight of the second portable information terminal is displayed on both the first and second portable information terminals. For the other virtual objects, images viewed from the line of sight of the first portable information terminal are displayed on both the first and second portable information terminals.

7. The image sharing system as described in claim 3, characterized in that: The first processor uses a virtual avatar corresponding to the user of the first portable information terminal to represent the viewpoint position of the first portable information terminal. The viewpoint position of the second portable information terminal is represented by a virtual avatar corresponding to the user of the second portable information terminal.

8. The image sharing system as described in claim 7, characterized in that: The first processor replaces the virtual avatars corresponding to the user of the first portable information terminal and the virtual avatars corresponding to the user of the second portable information terminal, and uses a common virtual avatar for viewing the individual display objects to represent the viewing position of the individual display objects.

9. An image sharing method, characterized in that, Perform the following steps: In the first step, the portable information terminal performs visual sharing control, switching the display mode of the virtual object selected as the visual sharing target among multiple users from the image viewed from the viewpoint of the portable information terminal to the image viewed from the viewpoint of other portable information terminals; and In the second step, the portable information terminal uses an image viewed from the viewpoint of the other portable information terminal to display the virtual object.

Images