Information processing device, information processing system, and information processing method

Abstract

This information processing device is provided with a sensor, a communication unit, and a circuit configuration. The sensor is configured so as to acquire relative position information of a main display device with respect to a sub-display device. The communication unit is configured so as to transmit viewpoint information of a user of the main display device to the sub-display device and / or receive viewpoint information of the user from the sub-display device. The circuit configuration is configured so as to: acquire the viewpoint information and the relative position information; acquire, on the basis of the relative position information, device absolute position information indicating position information about the main display device and the sub-display device in a coordinate system based on the main display device; and control display of the main display device on the basis of integrated information that is based on the viewpoint information and the device absolute position information.

Description

Information processing device, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method.

[0002] 2. Description of the Related Art Conventionally, a spatial display device is known as a type of naked-eye 3D (dimensional) display device.

[0003] The spatial display device detects the viewpoint of the user using a camera mounted on the device, generates a viewpoint image corresponding to this viewpoint, and displays it on a panel. The panel is equipped with a light distribution member such as a lenticular lens, which separates the image displayed on the panel into directions for the user's left and right eyes. This allows the user to view a 3D image with the naked eye.

[0004] In recent years, studies have been conducted on arranging a plurality of spatial display devices to display a wide-area scene in a coordinated manner, and a technique for realizing this has been proposed (see, for example, Patent Document 1).

[0005] International Publication No. 2022 / 230350

[0006] However, the above-mentioned conventional technology has room for further improvement in terms of enabling coordinated control including at least coordinated display using only a plurality of spatial display devices. For example, the above-mentioned conventional technology requires a large-scale backend, such as an imaging device and a signal processing device, for position detection in addition to the spatial display device.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing system, and an information processing method that enable cooperative control including at least cooperative display using only a plurality of spatial display devices.

[0008] In order to solve the above-described problems, an information processing device according to one embodiment of the present disclosure includes a sensor, a communication unit, and a circuit configuration. The sensor is configured to acquire relative position information of a main display device with respect to a sub-display device. The communication unit is configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device and receive viewpoint information of the user from the sub-display device. The circuit configuration is configured to acquire the viewpoint information and the relative position information, acquire device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device, and control display of the main display device based on integrated information based on the viewpoint information and the device absolute position information.

[0009] 15A and 15B are diagrams illustrating an example configuration of a display system according to an embodiment of the present disclosure; FIG. 16A is a diagram illustrating an example configuration of a spatial display device according to an embodiment of the present disclosure; FIG. 16B is a diagram illustrating a usage form of the display system; FIG. 16C is a diagram illustrating an example display expected in the display system; FIG. 16D is an explanatory diagram of the field of view of a typical human; FIG. 16E is a diagram illustrating a case where viewpoint acquisition is difficult in some spatial display devices; FIG. 16F is a diagram illustrating an example display in which some parts are illegal; FIG. 16F is a schematic explanatory diagram of an information processing method according to an embodiment of the present disclosure; FIG. 16H is a block diagram illustrating an example configuration of a spatial display device according to an embodiment of the present disclosure; FIG. 16I is a diagram illustrating a data flow when the target device is a main display device; FIG. 16I is a diagram illustrating a data flow when the target device is a sub-display device; FIG. 16J is a state transition diagram in a display system; FIG. 16I is an explanatory diagram (part 1) of the basic structure of a packet; FIG. 16I is an explanatory diagram (part 2) of the basic structure of a packet; FIG. 16I is a flowchart illustrating a processing procedure in an UNDER_CONNECTION state; and FIG. 15A is a processing sequence diagram corresponding to FIG. 15B. 23 and 24. FIG. 24 is a flowchart (part 1) showing a processing procedure as a sub-device in the TOPOLOGY_ESTABLISHED state. FIG. 25 is a flowchart (part 2) showing a processing procedure as a sub-device in the TOPOLOGY_ESTABLISHED state. FIG. 26 is a processing sequence diagram corresponding to FIGS. 17 to 21. FIG. 27 is a flowchart (part 3) showing a processing procedure as a sub-device in the TOPOLOGY_ESTABLISHED state. FIG. 28 is a processing sequence diagram corresponding to FIGS. 23 and 24. FIG. 29 is a diagram (part 1) showing a data structure in a display system according to an embodiment of the present disclosure. FIG. 29 is a diagram (part 2) showing a data structure in a display system according to an embodiment of the present disclosure. FIG. 30 is a diagram (part 3) showing a data structure in a display system according to an embodiment of the present disclosure.37 is a diagram showing a data structure (part 4) in a display system according to an embodiment of the present disclosure. FIG. 37 is a diagram showing an example of device layout. FIG. 37 is an explanatory diagram of relative position detection using UWB. FIG. 37 is a diagram showing an example of a UWB detection range. FIG. 37 is a diagram showing the detection results of localGeometry information of device 01. FIG. 37 is a diagram showing the detection results of localGeometry information of device 02. FIG. 37 is a diagram showing the detection results of localGeometry information of device 03. FIG. 37 is a diagram showing the detection results of localGeometry information of device 06. FIG. 37 is a diagram showing the detection results of localGeometry information of device 07. FIG. 37 is a diagram showing pairs of detection results and relative position information in FIGS. 33 to 37. FIG. 37 is a diagram showing the relationship between relative positions between devices and the detection results of localGeometry information. FIG. 37 is a diagram showing a relative position graph structure. FIG. 37 is a diagram showing the data structure of a graph structure. FIG. 37 is a diagram showing an example of graph data storage. FIG. 37 is a diagram showing the data structure of inter-device relative position information devPosRotEntry. FIG. 37 is a diagram showing initial branching in connection topology determination. 1 is a diagram showing an example of incidental conditions when determining a connection topology; FIG. 2 is a diagram showing connection topology determination conditions; FIG. 3 is a diagram showing a device viewpoint distance D; ｎ 1 is a diagram showing a derivation procedure of the above. A flowchart showing the processing procedure for connection topology determination processing. A diagram showing the data structure of connection topology information. A diagram showing the data structure of an integrated viewpoint. A diagram showing the data structure of shared information. An explanatory diagram for the case where two users are viewing facing each other. A diagram showing the initial state of LEP mode. A diagram showing the data structure of shared information related to a second modified example. A diagram showing the entry structure of faceDetectionList. An explanatory diagram (part 1) of an example of sensing relative positions using a combination of a light emitter and an RGB camera. An explanatory diagram (part 2) of an example of sensing relative positions using a combination of a light emitter and an RGB camera. An explanatory diagram of linked display using a device equipped with a face sensing unit and a device not equipped with a face sensing unit in combination. A hardware configuration diagram showing an example of a computer that realizes the functions of a spatial display device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] In the following description, when it is necessary to distinguish between multiple identical components, the reference numeral for the component may be followed by a number in the form of "-m" (where m is a natural number). When there is no particular need to distinguish between the components, this numbering will not be used.

[0012] In the following description, an information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is assumed to be a spatial display device 100 (see FIG. 1). An information processing system according to the present embodiment is assumed to be a display system 1 (see FIG. 1) including a plurality of spatial display devices 100. An information processing method according to the present embodiment is assumed to be executed by the spatial display device 100.

[0013] The present disclosure will be described in the following order: 1. Overview 1-1. Configuration example of a display system 1-2. Spatial display device 1-3. Usage form 1-4. Outline of information processing method according to an embodiment of the present disclosure 2. Configuration example of a spatial display device 3. State transition in a display system 4. Processing procedure 4-1. Basic structure of a packet 4-2. Processing procedure in the UNDER_CONNECTION state 4-3. Processing procedure in the TOPOLOGY_ESTABLISHED state 4-3-1. Main device 4-3-2. Sub-device 4-4. Data structure 4-5. Absolute position information 4-6. Connection topology information 4-7. Integrated viewpoint information 4-8. Shared information 5. Modifications 5-1. First modification 5-2. Second modification 5-3. Third modification 5-4. Fourth Modification 5-5. Other Modifications 6. Hardware Configuration 7. Conclusion

[0014] <<1. Overview>> <1-1. Configuration example of display system> Fig. 1 is a diagram showing a configuration example of a display system 1 according to an embodiment of the present disclosure. As shown in Fig. 1, the display system 1 includes a plurality of spatial display devices 100-1, 100-2, ..., 100-n (n is a natural number of 2 or more) and a wireless router 200.

[0015] The spatial display device 100 is a panel-integrated 3D image display device. The wireless router 200 is a repeater of the network N using wireless communication. The spatial display devices 100 are provided so as to be able to communicate with each other via the wireless router 200.

[0016] 2 is a diagram showing an example of the configuration of a spatial display device 100 according to an embodiment of the present disclosure. The spatial display device 100 has a control unit 101, a sensing unit 102, and a 3D panel display unit 103. The spatial display device 100 may further have a speaker (not shown) for audio output.

[0017] The control unit 101 is mounted inside the spatial display device 100, and generates a viewpoint image to be displayed on the 3D panel display unit 103 based on information acquired by the sensing unit 102 and information from a communication unit A5 (described later) included in the spatial display device 100. In the present disclosure, the viewpoint image may be regarded as an image having motion parallax based on the user's viewpoint position.

[0018] The sensing unit 102 is a component for detecting the relative position of the user's face and the other spatial display devices 100. For the former, a wide-angle camera is used, for example, and for the latter, a UWB (Ultra Wide Band) sensor is used. Information detected by the sensing unit 102 is sent to the control unit 101, where necessary processing is performed as appropriate.

[0019] The 3D panel display unit 103 is a panel-type light-emitting device equipped with a mechanism for separating and displaying light beams for the user's left and right eyes. The 3D panel display unit 103 is configured to display a stereoscopic image / video with binocular disparity generated by the control unit 101 via the mechanism for separating and displaying light beams. That is, in this embodiment, the viewpoint image has motion parallax and binocular parallax according to the user's face position or viewpoint. Note that the panel-type light-emitting device in the present disclosure is not limited to a configuration having a mechanism for separating and displaying light beams. That is, the viewpoint image may be a non-stereoscopic image without binocular parallax.

[0020] The spatial display device 100 is small and highly portable, and is realized as, for example, a smartphone-type or tablet-type device.

[0021] <1-3. Usage> Fig. 3 is a diagram showing a usage of the display system 1. The display system 1 realizes coordinated display of a wide-area scene using a plurality of spatial display devices 100. Because the spatial display device 100 is small and highly portable, it is expected that there will be an increasing number of display opportunities where a plurality of devices are deployed in various locations.

[0022] 3, the display system 1 arranges a plurality of spatial display devices 100 in a wide area in front of the eyes of the user U, and performs coordinated display of a wide-area scene. Each spatial display device 100 generates and displays a viewpoint video corresponding to the viewpoint position of the user U acquired therein.

[0023] 4 is a diagram showing an example of a display expected in the display system 1. The display system 1 displays, for example, a performance scene of a group of performers in cooperation as a wide-area scene. As shown in FIG. 4, the display system 1 displays each member of the group of performers on each spatial display device 100.

[0024] The spatial display device 100 can display images by positioning them with six degrees of freedom (6DoF) relative to the installation space. Therefore, in the spatial display devices 100-1 to 100-5 shown in Fig. 4, each performer appears to be performing as if they are actually standing on a stage, rather than in the direction of the user U's viewpoint.

[0025] This display can be realized by each spatial display device 100 acquiring the face position of the user U using the sensing unit 102 and generating an image viewed from that direction. However, in a method in which each spatial display device 100 individually senses the face position of the user U, there is a high possibility that display failure will occur in some spatial display devices 100 due to the relative positioning of the user U or the occurrence of unexpected occlusion.

[0026] 5 is an explanatory diagram of a typical human visual field. As shown in FIG. 5, a typical human visual field has a visual angle θ １ On the other hand, the range of 70° in front of the human is the visual field θ 1 , which allows distance estimation with binocular vision. ２ is.

[0027] That is, humans can perceive a stereoscopic display within a range of 70° in front of them. However, if the perceptual range of only one eye, either the left or right eye, is included, they can perceive changes in the field of view up to a maximum of approximately 200°. In other words, even if the spatial display devices 100 are arranged over a wide area, it is possible to detect changes in each display, which is thought to affect the comfort or discomfort of the viewing experience.

[0028] The angle of view of the camera used by each spatial display device 100 for face recognition is small compared to the field of view of the user U. For this reason, depending on the relative positional relationship between the spatial display device 100 and the user U, the face of the user U may fall outside the angle of view of the camera. It is also possible that the camera and the user U may be blocked due to some event during viewing.

[0029] Such a case is shown in Fig. 6. Fig. 6 is a diagram showing a case where viewpoint acquisition is difficult in some spatial display devices 100. Fig. 7 is a diagram showing an example of a display in which some of the spatial display devices 100 are incorrect.

[0030] 6, the spatial display device 100-1 cannot capture the user U in the viewing angle of the camera. Also, the spatial display device 100-3 cannot correctly detect the face of the user U due to an obstruction (e.g., a hand) passing in front of the camera.

[0031] If the information processing method according to this embodiment is not applied, the spatial display devices 100-1 and 100-3 cannot correctly know the face position of the user U. For this reason, as shown in Fig. 7, the spatial display devices 100-1 and 100-3 will display an unnatural image that is not spatially localized.

[0032] <1-4. Overview of Information Processing Method According to Embodiment of the Present Disclosure> Fig. 8 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. In the information processing method according to the present embodiment, as shown in Fig. 8, one of the spatial display devices 100 is designated as a "main display device" that is a master device, and the others are designated as "sub-display devices" that are slave devices.

[0033] 8, the spatial display device 100-2 is the main display device "M," and the other spatial display devices 100-1, 100-3 to 100-5 are sub-display devices "S." In the information processing method according to this embodiment, the spatial display device 100-1 integrates the position information and viewpoint information of all the sub-display devices "S" arranged in the vicinity into shared information, and distributes this information to each of the sub-display devices "S."

[0034] Here, the spatial display devices 100-4 and 100-5 are sub-display devices "S" capable of acquiring viewpoints, which provide their own viewpoint information to the main display device "M" and receive shared information integrated by the main display device "M".

[0035] On the other hand, the spatial display devices 100-1 and 100-3 are sub-display devices "S" that have difficulty in obtaining a viewpoint, and provide a status (obtaining is not possible) regarding viewpoint obtaining to the main display device "M", and receive shared information from the main display device "M".

[0036] The roles of the main display device "M" and the sub-display device "S" are changed in real time between the spatial display devices 100 in response to changes in the surrounding situation, operations by the user U, etc. This mechanism makes it possible to display viewpoint images that are correctly positioned in space on all spatial display devices 100, as shown in Figure 4, even in a situation where face sensing of the user U is obstructed on some of the spatial display devices 100.

[0037] Therefore, the information processing method according to this embodiment makes it possible to perform coordinated control including at least coordinated display using only a plurality of spatial display devices 100. An example configuration of the display system 1 to which the information processing method according to this embodiment is applied will be described in more detail below.

[0038] <<2. Configuration Example of Spatial Display Device>> Fig. 9 is a block diagram showing a configuration example of a spatial display device 100 according to an embodiment of the present disclosure. As shown in Fig. 9, the spatial display device 100 has a control unit 101, a sensing unit 102, and a 3D panel display unit 103. Hereinafter, the spatial display device 100 that is the subject of the description may be referred to as the "target device" as appropriate, and other spatial display devices 100 may be referred to as "other devices."

[0039] The control unit 101 has a face sensing unit A1, a relative position sensing unit A2, an information processing unit A3, and a video processing unit A4. The face sensing unit A1 is a block that acquires an image sequence captured by a camera of the sensing unit 102, which is attached in the same direction as the 3D panel display unit 103, in order to acquire the face position of the user U, and transmits the image sequence to the information processing unit A3.

[0040] The relative position sensing unit A2 is a block including a sensor for acquiring the relative positional relationship between the spatial display devices 100 and a device for transmitting and receiving the sensing results. Since the present embodiment is assumed to use a UWB sensor, a UWB anchor device may also be used.

[0041] The information processing unit A3 is a block that calculates viewpoint information used by the video processing unit A4 and information to be transmitted to other devices by the communication unit A5, based on information transmitted from the face sensing unit A1 and the relative position sensing unit A2, and information of other spatial display devices 100 (hereinafter referred to as "other devices" as appropriate) acquired by the communication unit A5. The behavior of the information processing unit A3 differs depending on the master / slave relationship of the spatial display device 100.

[0042] The video processing unit A4 determines the camera position from the viewpoint information received from the information processing unit A3, performs rendering processing of the specified content and surrounding scene data, and generates a viewpoint video for panel display. The 3D panel display unit 103 receives and displays the viewpoint video generated by the video processing unit A4.

[0043] The communication unit A5 packetizes the shared information generated by the information processing unit A3 and transmits it. The communication unit A5 is, for example, a transmitter. It also receives packets transmitted from the network N to the spatial display device 100 (hereinafter referred to as the "target device"), extracts necessary information, and passes it to the information processing unit A3.

[0044] Next, a description will be given of the data flow of the spatial display device 100. Fig. 10 is a diagram showing the data flow when the target device is a main display device. Fig. 11 is a diagram showing the data flow when the target device is a sub-display device.

[0045] 10 and 11, each functional block is shown as a white block, and information exchanged between functional blocks is shown as a shaded block.

[0046] 10, when the target device is the main display device, the face sensing unit A1 functions as a face sensing unit C1 and handles target device face sensing information D1, and the relative position sensing unit A2 functions as a relative position sensing unit C2 and handles target device relative position D2.

[0047] The information processing unit A3 functions as a connection topology control unit C3, an integrated viewpoint information generation unit C4, a device position coordinate conversion unit C5, a shared information generation unit C6, and a slave device information analysis unit C8. The information processing unit A3 also handles connection topology information D3, integrated viewpoint information D4, other device / face sensing information D5, other device relative positions D6, and device absolute position information D7.

[0048] The video processing unit A4 includes a video generation unit C7 and a video display unit C9, which operate to generate a panel video D8. The communication unit A5 includes a slave device information receiving unit C10 and a shared information transmitting unit C11, which operate to generate slave device information D9 and shared information D10.

[0049] On the other hand, when the target device is a sub-display device, as shown in Fig. 11, the face sensing unit A1 functions as a face sensing unit C1 and handles target device face sensing information D1, and the relative position sensing unit A2 functions as a relative position sensing unit C2 and handles target device relative position D2.

[0050] The information processing unit A3 functions as a shared information analysis unit C12 and a slave device information generation unit C13, and handles connection topology information D3 and integrated viewpoint information D4.

[0051] The video processing unit A4 includes a video generation unit C7 and a video display unit C9, and handles a panel video D8. The communication unit A5 includes a slave device information transmission unit C14 and a shared information reception unit C15, and handles slave device information D9 and shared information D10.

[0052] When the target device is a main display device, as shown in FIG. 10, the face sensing unit C1 generates target device face sensing information D1, which is received by the connection topology control unit C3 and the integrated viewpoint information generation unit C4.

[0053] The relative position sensing unit C2 generates a target device relative position D2, which is received by a device position coordinate conversion unit C5. The connection topology control unit C3 generates connection topology information D3 based on the other device / face sensing information D5 and the device absolute position information D7, which is received by a shared information generation unit C6. The connection topology information D3 is information related to topology management of the main display device and sub-display device in the network N of the display system 1.

[0054] The integrated viewpoint information generator C4 generates integrated viewpoint information D4, which is received by the shared information generator C6 and the video generator C7. The device position coordinate converter C5 generates device absolute position information D7 based on the target device relative position D2 and the other device relative position D6, which is received by the connection topology controller C3 and the shared information generator C6.

[0055] The shared information generation unit C6 generates shared information D10 based on the connection topology information D3, the integrated viewpoint information D4, and the device absolute position information D7, and the shared information transmission unit C11 receives this. The video generation unit C7 generates a panel video D8 based on the target device / face sensing information D1 and the integrated viewpoint information D4, and the video display unit C9 receives this and performs viewpoint video processing for the 3D panel display unit 103.

[0056] The slave device information analysis unit C8 generates the other device relative position D6 based on the slave device information D9, and this is received by the device position coordinate conversion unit C5. The slave device information reception unit C10 receives the slave device information D9 from the sub-display device via the network N and passes it to the slave device information analysis unit C8. The shared information transmission unit C11 transmits the shared information D10 to the sub-display device via the network N.

[0057] When the target device is a sub-display device, as shown in FIG. 11, the face sensing unit C1 generates target device face sensing information D1, which is received by the video generation unit C7 and the slave device information generation unit C13.

[0058] The relative position sensing unit C2 generates the target device relative position D2, which is received by the slave device information generation unit C13. The image generation unit C7 generates a panel image D8 based on the target device / face sensing information D1 and the integrated viewpoint information D4, and the image display unit C9 receives this and performs viewpoint image processing for the 3D panel display unit 103.

[0059] The shared information analysis unit C12 analyzes the shared information based on the shared information D10 and outputs the analysis results to the connection topology information D3 and the integrated viewpoint information D4. The slave device information generation unit C13 generates slave device information D9 based on the target device / face sensing information D1 and the target device relative position D2, and passes it to the slave device information transmission unit C14.

[0060] The slave device information transmitting section C14 transmits the slave device information D9 to the main display device via the network N. The shared information receiving section C15 receives the shared information D10 from the main display device via the network N and passes it to the shared information analyzing section C12.

[0061] <<3. State Transition in Display System>> Next, a description will be given of state transition in the display system 1. FIG.

[0062] As shown in Fig. 12, in the display system 1, the spatial display device 100 has two states: "UNDER_CONNECTION" and "TOPOLOGY_ESTABLISHED". The UNDER_CONNECTION state indicates a state in which the master and slave of each spatial display device 100 in the display system 1 has not been determined. The TOPOLOGY_ESTABLISHED state indicates a state in which at least the master device has been determined in the display system 1. ｎ indicates a transition between states, and transitions occur according to the conditions shown in FIG.

[0063] <<4. Processing Procedure>> <4-1. Basic Structure of Packet> Next, the processing procedure in each state shown in Fig. 12 will be explained in order using Figs. 13 to 25. First, the basic structure of a packet exchanged between each spatial display device 100 in the display system 1 will be explained. Fig. 13 is an explanatory diagram (part 1) of the basic structure of a packet. Also, Fig. 14 is an explanatory diagram (part 2) of the basic structure of a packet.

[0064] In the following description, the main display device may be referred to as the "main device," and the sub-display device may be referred to as the "sub-device."

[0065] As shown in FIG. 13, packets exchanged in the display system 1 include requests (REQ) having four types of "msgType" and responses (RES) having three types of "rspType".

[0066] "MAIN_DISCOVERY" with msgType "001" corresponds to REQ1. REQ1 is a discovery request for discovering a main device. REQ1 corresponds to an example of a "first request inquiring about the presence or absence of a main display device."

[0067] "SUB_REGISTRATION" with msgType "002" corresponds to REQ2. REQ2 is a sub-device registration request, which is a request to register as a sub-device. REQ2 corresponds to an example of a "second request."

[0068] "STATUS_REPORT" with msgType "003" corresponds to REQ 3. REQ 3 is a report on the status of the sub-device sent from the sub-device to the main device.

[0069] "STATUS_PUSH" with msgType "004" corresponds to REQ4. REQ4 is information sent from the main device to the sub-device.

[0070] Figure 14 shows, from top to bottom, the data structure for each pair of REQ1 to REQ4 and RES1 to RES4 corresponding to REQ1 to REQ4. As shown in Figure 14, REQ3 includes slave device information D9 as a report on the status of the sub-device. REQ4 also includes shared information D10 that is generated periodically or as needed by the main device. The processing procedure for exchanging these packets will be described below.

[0071] 4-2. Processing Procedure in UNDER_CONNECTION State First, the processing procedure in the UNDER_CONNECTION state will be described. Fig. 15 is a flowchart showing the processing procedure in the UNDER_CONNECTION state. Fig. 16 is a processing sequence diagram corresponding to Fig. 15.

[0072] When the spatial display device 100 is in the UNDER_CONNECTION state, the control unit 101 of the spatial display device 100 first acquires the target device face sensing information D1 and the target device relative position D2 (step S101).

[0073] Then, the control unit 101 broadcasts a discovery request REQ1 (step S102), and then determines whether or not a response RES1 has been received from the main device (step S103).

[0074] If the response RES1 is received (step S103, Yes), the control unit 101 generates slave device information D9 from the target device / face sensing information D1 and the target device relative position D2 (step S104).Then, the control unit 101 sends a sub-device registration request REQ2 to the main device (step S105).

[0075] Then, after waiting for acceptance by the main device (step S106), the control unit 101 determines whether or not a response RES2 has been received (step S107). If the response RES2 has not been received (step S107, No), the control unit 101 repeats the process from step S106. If the response RES2 has been received (step S107, Yes), the control unit 101 changes the state to TOPOLOGY_ESTABLISHED (step S108) and ends the process.

[0076] On the other hand, if the response RES1 is not received in step S103 (step S103, No), the control unit 101 designates the target device as the main device and causes the target device to perform tasks as the main display device (step S109). Tasks as the main display device include, for example, generating connection topology information D3, sending a response indicating that the main device has already been configured, registering sub-devices in the connection topology information D3, updating the connection topology information D3, and generating and updating shared information D10. Then, the control unit 101 converts the target device relative position D2 into device absolute position information D7 using the device position coordinate conversion unit C5 (step S110).

[0077] The control unit 101 then passes the target device / face sensing information D1 and the device absolute position information D7 to the connection topology control unit C3, and generates connection topology information D3 (step S111). The control unit 101 then changes the state to TOPOLOGY_ESTABLISHED (step S108), and ends the process. Note that a processing sequence diagram corresponding to the processing procedure of FIG. 15 is as shown in FIG. 16.

[0078] <4-3. Processing Procedures in TOPOLOGY_ESTABLISHED State> (4-3-1. Main Device) Next, the processing procedures as the main device in the TOPOLOGY_ESTABLISHED state will be described. Figures 17 to 21 are flowcharts (parts 1) to (part 5) showing the processing procedures as the main device in the TOPOLOGY_ESTABLISHED state. Figure 22 is a processing sequence diagram corresponding to Figures 17 to 21.

[0079] To give an overview of this process, the basic process is to receive a request from a sub-device in a message processing loop, perform the necessary processing, and send the result to the sub-device that made the request. Also, in the message processing loop, shared information D10 is generated at regular intervals and PUSH_STATUS is sent to all sub-devices.

[0080] When generating the shared information D10, the connection topology information D3 is recalculated, but due to changes in the conditions of the installation environment, a mainDeviceId different from that of the target device may be selected. In that case, the target device is changed to a sub-device, the flow as the main device is terminated, and the processing procedure is started again in the new sub-device state of TOPLOGY_ESTABLISHED. Waiting to receive request REQ4 is an abnormal process, and if it is received, DUPLICATED_MAIN is returned as an error.

[0081] Specifically, as shown in FIG. 17, when the state transitions from the UNDER_CONNECTION state to the TOPOLOGY_ESTABLISHED state, the control unit 101 starts a message processing loop for the packet (step S201).

[0082] In this message processing loop, the control unit 101 determines whether or not the request REQ1 has been received (step S202). If the request REQ1 has been received (step S202, Yes), the control unit 101 executes MAIN_DISCOVERY processing (step S203), and then proceeds to step S210.

[0083] If the request REQ1 has not been received (step S202, No), the control unit 101 determines whether the request REQ2 has been received (step S204). If the request REQ2 has been received (step S204, Yes), the control unit 101 executes SUB_REGISTRATION processing (step S205), and then proceeds to step S210.

[0084] If the request REQ2 has not been received (step S204, No), the control unit 101 determines whether the request REQ3 has been received (step S206). If the request REQ3 has been received (step S206, Yes), the control unit 101 executes STATUS_REPORT processing (step S207), and then proceeds to step S210.

[0085] If the control unit 101 has not received the request REQ3 (No at step S206), it determines whether or not the control unit 101 has received the request REQ4 (step S208). If the control unit 101 has received the request REQ4 (Yes at step S208), it transmits DUPLICATED_MAIN to the sender of the request REQ4 (step S209), and then proceeds to step S210.

[0086] If the request REQ4 has not been received (No at step S208), the control unit 101 proceeds directly to step S210. At step S210, the control unit 101 executes STATUS_PUSH processing #1 (step S210).

[0087] The control unit 101 then performs viewpoint video processing in the video processing unit A4 based on the viewpoint information of the target device / face sensing information D1 and the integrated viewpoint information D4 (step S211). Then, it determines whether the message processing loop has ended (step S212). If the message processing loop has not ended (step S212, No), the process is repeated from step S202. If the message processing loop has ended (step S212, Yes), the process is terminated.

[0088] Next, the MAIN_DISCOVERY process will be described. As shown in Fig. 18, in the MAIN_DISCOVERY process, the control unit 101 sets the displayId of the target device to the mainDisplayId of the packet (step S301).

[0089] Then, the control unit 101 transmits a response RES1 to the sender of the request REQ1 (step S302), and ends the MAIN_DISCOVERY process.

[0090] The MAIN_DISCOVERY process will now be described. As shown in Fig. 18, in the MAIN_DISCOVERY process, the control unit 101 sets the displayId of the target device to the mainDisplayId of the packet (step S301).

[0091] Then, the control unit 101 transmits a response RES1 to the sender of the request REQ1 (step S302), and returns from the MAIN_DISCOVERY process.

[0092] Next, the SUB_REGISTRATION process will be described. As shown in Fig. 19, in the SUB_REGISTRATION process, the control unit 101 adds the displayId of the request REQ2 to the displayId list (step S401).

[0093] Then, the control unit 101 transmits a response RES2 to the sender of the request REQ2 (step S402), and returns from the SUB_REGISTRATION process.

[0094] Next, the STATUS_REPORT process will be described. As shown in Fig. 20, in the STATUS_REPORT process, the control unit 101 updates the entry in the deviceStatusList corresponding to the displayId of the request REQ3 with the slave device information D9 of the request REQ3 (step S501).

[0095] Then, the control unit 101 transmits a response RES3 to the sender of the request REQ3 (step S502), and returns from the STATUS_REPORT process.

[0096] 21, in STATUS_PUSH processing #1, the control unit 101 inputs deviceStatusList to the information processing unit A3, and outputs connection topology information D3, integrated viewpoint information D4, and device absolute position information D7 (step S601).

[0097] Then, the control unit 101 generates shared information D10 from the connection topology information D3, the integrated viewpoint information D4, and the device absolute position information D7 (step S602), and stores it in the response RES4 (step S603).

[0098] Then, the control unit 101 broadcasts a request REQ4 to the destinations in the displayId list (step S604).Then, the control unit 101 determines whether the mainDeviceId of the connection topology information D3 is the deviceId of the target device (step S605).

[0099] If the deviceId is not that of the target device (Yes at step S605), the control unit 101 sets the target device as a sub-device, transitions to a TOPOLOGY_ESTABLISHED state (step S606), and ends the process.

[0100] If the deviceId is the deviceId of the target device (step S605, No), the control unit 101 determines whether the resType of the response RES4 is DUPLICATED_MAIN (step S607).

[0101] If the state is DUPLICATED_MAIN (step S607, Yes), the control unit 101 transitions to the UNDER_CONNECTION state and ends the process. If the state is not DUPLICATED_MAIN (step S607, No), the control unit 101 returns from STATUS_PUSH process #1. Note that the process sequence diagram corresponding to the process procedures of FIGS. 17 to 21 is as shown in FIG. 22.

[0102] (4-3-2. Sub-Device) Next, the processing procedure as a sub-device in the TOPOLOGY_ESTABLISHED state will be described. Figures 23 and 24 are flowcharts (part 1) and (part 2) showing the processing procedure as a sub-device in the TOPOLOGY_ESTABLISHED state. Figure 25 is a processing sequence diagram corresponding to Figures 23 and 24.

[0103] To give an overview of this process, if the mainDeviceId of the connection topology information D3 included in the shared information D10 matches the deviceId of the target device, it means that the main device has determined that the target device has been designated as the new main device. The fact that the mainDeviceId of the connection topology information D3 matches the deviceId of the target device indicates that the shared information D10 includes identification information (the deviceId of the target device) indicating the main display device. In this case, the role of the target device is changed to the main device, the task as a sub-device is terminated, and a new task as a main device in the TOPOLOGY_ESTABLISHED state is started.

[0104] If STATUS_PUSH is not received from the main device for a certain period of time, it is assumed that the main device is unable to continue processing for some reason, and the state is changed to UNDER_CONNECTION.

[0105] Specifically, as shown in FIG. 23, when the state transitions from the UNDER_CONNECTION state to the TOPOLOGY_ESTABLISHED state, the control unit 101 starts a message processing loop for the packet (step S701).

[0106] In this message processing loop, the control unit 101 inputs the target device and face sensing information D1 and the target device relative position D2 to the slave device information generation unit C13, and generates slave device information D9 (step S702).

[0107] Then, the control unit 101 generates a request REQ3 from the slave device information D9 and transmits it to mainDisplayId (i.e., the main device) (step S703). Subsequently, the control unit 101 determines whether or not a request REQ4 has been received (step S704).

[0108] If the request REQ4 is received (step S704, Yes), the control unit 101 executes STATUS_PUSH process #2 (step S705), and then proceeds to step S708. If the request REQ4 is not received (step S704, No), the control unit 101 increments the counter (step S706), and determines whether the counter exceeds the threshold value (step S707).

[0109] If the counter is below the threshold value (step S707, No), the control unit 101 performs viewpoint image processing in the image processing unit A4 based on the viewpoint information of the target device / face sensing information D1 and the integrated viewpoint information D4 (step S708).Then, it determines whether the message processing loop has ended (step S709).If it has not ended (step S709, No), the processing is repeated from step S702.If it has ended (step S709, Yes), the processing is terminated.

[0110] On the other hand, if the counter exceeds the threshold value (Yes at step S707), the control unit 101 transitions to the UNDER_CONNECTION state (step S710) and ends the process.

[0111] 24, in STATUS_PUSH processing #2, the control unit 101 processes the shared information D10 of the request REQ4 in the shared information analysis unit C12, and extracts connection topology information D3 and integrated viewpoint information D4 (step S801).

[0112] Then, the control unit 101 determines whether the mainDeviceId of the connection topology information D3 is the deviceId of the target device (step S802).

[0113] If the deviceId is that of the target device (Yes at step S802), the control unit 101 sets itself as the main device, transitions to the TOPOLOGY_ESTABLISHED state (step S803), and ends the process.

[0114] If the deviceId is not its own (step S802, No), the control unit 101 resets the counter (step S804) and returns from STATUS_PUSH process #2. Note that a process sequence diagram corresponding to the process procedures of Figs. 23 and 24 is as shown in Fig. 25.

[0115] 26 to 29 show data structures in the display system 1 according to the embodiment of the present disclosure. 26 to 29 are diagrams (part 1) to (part 4) showing data structures in the display system 1 according to the embodiment of the present disclosure.

[0116] FIG. 26 shows the data structure of viewpoint information faceDetection acquired by each sub-device or integrated by the main device.

[0117] 27 shows the data structure of the localGeometry, which is the relative position of another device acquired by each spatial display device 100. Note that if two or more other devices are placed within the detectable range around the device, multiple results of this data will be acquired.

[0118] 28 shows the data structure of the slave device information D9 (subDeviceInfo) sent by each sub device to the main device. The localGeometryList at the end has the relative position localGeometry of the other device as an entry.

[0119] FIG. 29 shows the data structure of each device status, deviceStatus, which is an entry in the deviceStatusList managed by the main device.

[0120] 4-5. Absolute Position Information Next, we will describe the processing procedure for determining the device absolute position information D7 from the local geometry information acquired by each spatial display device 100. First, we will describe the mechanism by which each sub-device acquires local geometry information.

[0121] 30 is a diagram showing an example of device arrangement. Note that, in the following, depending on the deviceId of each spatial display device 100, if the deviceId is 01, it may be referred to as "device 01".

[0122] FIG. 30 shows an example in which sub-device 01 is placed to the front left of sub-device 05, and sub-device 03 is placed to the front right.

[0123] As mentioned above, each sub-device uses the UWB protocol to detect the relative position of other devices. Figure 31 is an explanatory diagram of relative position detection using UWB. Figure 31 shows a situation in which the distance and direction to other devices are detected using the corresponding UWB protocol.

[0124] Each device acquires the distance to other devices using the UWB Time of Arrival (ToA) protocol or the Time Difference of Arrival (TDoA) protocol, and then acquires the direction of arrival of the detection wave as a relative angle to the normal to the sensor surface using the Angle of Arrival (AoA) protocol.

[0125] By obtaining the distance and direction, the relative position vector for the target device is uniquely determined. Furthermore, since this detection result is performed bidirectionally for a symmetrical device pair, the detection results of both are combined to obtain relative position information M, which is a relative position matrix from device P to device Q. ＲＴ＿Ｐ＿Ｑ is obtained.

[0126] The range of location detection using UWB is limited to a radius of about 0.5 m around the device. Therefore, when a large number of devices are installed, the detection results will be localized. Figure 32 shows an example of a UWB detection range.

[0127] Figure 32 shows a case where the detection ranges are spread out enough that they do not overlap. In Figure 32, 12 devices are placed in an area that is approximately 2.0 m wide and 1.0 m long. As mentioned above, the detection range of a UWB device is a circle with a radius of approximately 0.5 m centered on each device, so it can be seen that devices placed in the right and left halves of an area cannot directly detect devices placed in the other area.

[0128] From here on, the following processing procedure will be explained assuming that only five devices, namely, device 01, device 02, device 03, device 06, and device 07, are located in the upper left part of Fig. 32. The expected detection results for each device are shown in Figs. 33 to 37.

[0129] Fig. 33 is a diagram showing the detection results of local geometry information for device 01. Fig. 34 is a diagram showing the detection results of local geometry information for device 02. Fig. 35 is a diagram showing the detection results of local geometry information for device 03. Fig. 36 is a diagram showing the detection results of local geometry information for device 06. Fig. 37 is a diagram showing the detection results of local geometry information for device 07.

[0130] Also, Fig. 38 is a diagram showing pairs of detection results and relative position information in Fig. 33 to Fig. 37. As described above, detection is performed in both directions, so as shown in Fig. 38, pairs of detection results from device P to device Q and detection results from device Q to device P exist in all of the detection results in Fig. 33 to Fig. 37.

[0131] In the example of FIG. 38, the relative position information M ＲＴ＿１＿２ , M ＲＴ＿１＿６ , M ＲＴ＿２＿３ , M ＲＴ＿２＿６ , M ＲＴ＿２＿７ , M ＲＴ＿３＿７ , M ＲＴ＿６＿７ is obtained.

[0132] Relative position information M ＲＴ＿Ｐ＿Ｑ is information indicating the relative translation and rotation movement of device Q when viewed from device P, and is expressed as a 4x4 matrix or a combination of a three-dimensional vector and a quaternion. A specific method for deriving the relative position (translation and rotation) is described below.

[0133] 39 is a diagram showing the relationship between the relative positions between devices and the detection results of localGeometry information. The coordinate systems xy and x'y' are device-specific coordinate systems for devices P and Q, respectively, with O and O' being their respective origins. The localGeometry information detected by the position sensor of each device, i.e., the position and orientation of other devices, are values ​​in this coordinate system.

[0134] Since the position detection is mutual, in FIG. 38, the position vector V Ｐ＿Ｑ and the position vector V where device Q detects device P. Ｑ＿Ｐ are the same vectors with different directions in the absolute coordinate system. On the other hand, the rotation angle R ＿Ｐ , R ＿Ｑ will have different values.

[0135] Next, V in the coordinate system xyz Ｐ＿Ｑ Rotation matrix M in the direction Ｒ＿Ｐ The rotation matrix can be generated from the unit vectors of the Cartesian coordinate axes, and the y-axis is V Ｐ＿Ｑ and the z-axis is set to the upward direction on the paper, the following equations (a) to (d) can be used to calculate the angle.

[0136]

[0137] In this example, it is assumed that all devices are placed on the same plane, such as a table, and that the z-axis of the device coordinate system is common. If all three axes (x, y, and z) are displaced for each device, it is necessary to calculate the z-axis as well, for example, by referencing the values ​​of the inertial sensor (IMU: Inertial Measurement Unit) equipped in each device.

[0138] Similarly, -V in the coordinate system x'y'z' Ｑ＿Ｐ Rotation matrix M in the direction Ｒ＿Ｑ is calculated using the following formulas (e) to (h).

[0139]

[0140] Here, V Ｐ＿Ｑ The reason is that V Ｐ＿Ｑ Ha-V Ｑ＿Ｐ and in the absolute coordinate system, but have different values ​​in the device-specific coordinate system.

[0141] And the rotation M of the x'y'z' coordinate system relative to the xyz coordinate system Ｒ＿Ｐ＿Ｑ is M Ｒ＿Ｑ and M Ｒ＿Ｑ It is calculated by the following formula (i) using

[0142]

[0143] Furthermore, the translational component V Ｐ＿Ｑ The x, y, and z components of ｘ , t ｙ , t ｚ , M Ｒ＿Ｐ＿Ｑ The m-th row and n-th column elements of ｍｎ Then, the final 4x4 rotation and translation matrix M ＲＴ＿Ｐ＿Ｑ can be written as the following equation (j).

[0144]

[0145] When these calculations are performed for all pairs, a set of relative position matrices corresponding to a graph structure such as that shown in Figure 40 becomes available. Figure 40 is a diagram showing a relative position graph structure. In the graph structure of Figure 40, even when devices that are the start and end points are determined, there may be multiple routes.

[0146] 40, there are four routes (06-01-02, 06-02, 06-07-02, 06-07-03-02) from device 06 to device 02. When generating device absolute position information D7, errors between these routes are removed by selection or optimization.

[0147] The device absolute position information D7 is the position (translation and rotation) information of each device based on the device-specific coordinate system of the main device. This is obtained by tracing the route from the main device to each device in order on the graph of FIG. 40, and obtaining the relative position information M ＲＴ＿Ｐ＿Ｑ It can be found by accumulating the above.

[0148] Fig. 41 is a diagram showing the data structure of a graph structure, Fig. 42 is a diagram showing an example of how graph data is stored, and Fig. 43 is a diagram showing the data structure of inter-device relative position information devPosRotEntry.

[0149] The relative position graph structure of Fig. 40 is packed into the data format of graphStructure shown in Fig. 41 and stored in connection topology information D3, which will be described later. As an example, the data contents of graphStructure when the graph of Fig. 40 is stored are shown in Fig. 42.

[0150] Furthermore, the relative positional relationship corresponding to each edge in Fig. 40 is expressed by inter-device relative position information devPosRotEntry of device absolute position information D7 shown in Fig. 43. This data is stored in the shared information D10 in a list format having the same number of entries as nEdge in graphStructure in Fig. 41.

[0151] <4-6. Connection Topology Information> Next, the connection topology information D3 will be described. The connection topology information D3 is information regarding the main / sub assignment of each spatial display device 100. Fig. 44 is a diagram showing initial branching in determining the connection topology.

[0152] Figure 44 shows a rough flow of state branching from the initial state. As shown in Figure 44, after a target device broadcasts a discovery request and waits a certain period of time for a response, consider two cases: 1) when only the target device is present, and 2) when a main device is already present.

[0153] If there is only the target device, it is assumed that the target device will be used alone or that it is the first device in the process of being deployed. In this case, the target device is set as the main device, and the process transitions to TOPLOGY_ESTABLISHED and ends.

[0154] On the other hand, if a main device already exists, the latest status of the target device is packed and a SUB_REGISTRATION is issued to the main device to register it as a sub-device. After receiving a reply indicating that registration is complete, the target device is set as a sub-device and the process transitions to TOPOLOGY_ESTABLISHED to end.

[0155] Even after the main / sub assignment is performed in the initial transition, the main device updates the connection topology information D3 at regular intervals. The update process of the connection topology information D3 will be described in detail below.

[0156] First, assume that five devices, device 01, device 02, device 03, device 06, and device 07, are deployed, each in a different state with respect to the environment and user U. Fig. 45 is a diagram showing an example of incidental conditions when determining a connection topology.

[0157] In the example of Fig. 45, there is an obstruction (hand) between device 01 and user U, which is an obstacle to the sensing of the face position of user U. In addition, strong lighting is shining on devices 02 and 03, which may also affect the sensing of the face position of user U. Taking these incidental circumstances into consideration, Fig. 46 shows the relevant parameters of each device that are referenced in determining the connection topology. Fig. 46 is a diagram showing the connection topology determination conditions.

[0158] For the sub-devices, "Status" and "Credibility" are obtained from "subDeviceInfo.faceDetection" included in the slave device information D9 that each device sends to the main device in the request REQ3 that is "STATUS_REPORT." The main device obtains the same information from the target device face sensing information D1.

[0159] "D ｎ " is the distance from the origin of the device-specific coordinates of the target device to the viewpoint of the user U (device viewpoint distance), and is calculated by the procedure shown in FIG. 47. ｎ . ｐｏｓ＿６ will be the origin (0,0,0) if device 06 is the main device.

[0160] The contents of the connection topology determination conditions shown in FIG. 46 are managed by the connection topology control unit C3 as a deviceStatusList having the device status deviceStatus shown in FIG. 29 as an entry.

[0161] Specifically, the connection topology control unit C3 generates this when the target device starts its role as a main device and discards it when the role ends. Also, every time the target device receives a STATUS_REPORT from a sub-device, the entry for the corresponding deviceId is recalculated and updated.

[0162] Using the deviceStatusList managed and updated as described above, the connection topology control unit C3 asynchronously generates connection topology information D3. The processing procedure for this processing is shown in Fig. 48. Fig. 48 is a flowchart showing the processing procedure for connection topology determination processing. Fig. 49 is a diagram showing the data structure of the connection topology information D3.

[0163] The connection topology control unit C3 first reads out the deviceStatusList (step S901), and then excludes devices whose Status is False (NG) (step S902).

[0164] Then, it is determined whether the number of entries is 0 (step S903). If no target device remains at this point, an error message indicating that there are no valid devices is displayed. Specifically, if the number of entries is 0 in step S903 (step S903, Yes), the connection topology control unit C3 displays an error message such as "No valid device was found" (step S904).

[0165] The connection topology control unit C3 then determines whether to terminate the process (step S905) and confirms whether to continue the process. If the process is not to be terminated (step S905, No), the connection topology control unit C3 waits for a certain period of time (step S906), and then repeats the process from step S901. If the process is to be terminated (step S905, Yes), the use of the device ends at that point.

[0166] If the number of entries is not 0 in step S903 (No in step S903), that is, if a valid entry exists, the connection topology control unit C3 calculates a sort key S for each entry. ｎ (Step S907). The connection topology control unit C3 calculates the sort key S by the following formula (k): ｎ Calculate.

[0167]

[0168] As shown in Figure 45, a hand is placed in a way that blocks the camera's field of view for device 01. In addition, strong illumination light is pouring onto the panel surfaces of devices 02 and 03, which is thought to have led to a decrease in reliability.

[0169] "D ｎ When the above formula (k) is used to calculate the valid entries (Status = OK) in FIG. 46 using ", " and "Credibility", the sort key S ｎ In this case, the connection topology control unit C3 selects the device 06 as the main device.

[0170] The connection topology control unit C3 basically selects the device that is capable of sensing the user U and is closest to the user U as the main device, but by weighting it based on reliability, it lowers the priority of devices that are "close but have low facial recognition accuracy."

[0171] Then, the connection topology control unit C3 assigns the deviceId of the device finally selected as the main device to mainDeviceId, and stores it together with the graphStructure in FIG. 41 in the connection topology information D3 shown in FIG.

[0172] That is, as shown in FIG. 48, after executing step S907, the connection topology control unit C3 assigns the entry to the key S ｎ The entries are sorted in descending order (step S908).The connection topology control unit C3 then sets the deviceId of the first entry after sorting to the maindeviceId (step S909), and ends the process.

[0173] <4-7. Integrated viewpoint information> Next, the integrated viewpoint information D4 will be described. Fig. 50 is a diagram showing the data structure of the integrated viewpoint information D4. For the integrated viewpoint information D4, the control unit 101 (integrated viewpoint information generation unit C4) first converts the acquired viewpoints of each device into a coordinate system reference for the main device using a graph matrix set.

[0174] The face coordinate vector before transformation of device n is faceVec ｎ , after transformation, faceVec ｎ ’ Then, the calculations in the devices 01, 02, 03, 06, and 07 are as shown in the following equations (l) to (p).

[0175]

[0176] Furthermore, the obtained viewpoint is weighted by the face detection credibility of each device n. ｉ Taking a weighted average using the following equation (q), the integrated viewpoint faceVec overline (upper line) is obtained.

[0177]

[0178] The integrated viewpoint faceVec overline and all M ＲＴ＿Ｐ＿Ｑ is STATUS_PUSHed to each device as shared information D10, and can be returned to device-specific coordinates on each device side using the following equations (r) to (v).

[0179]

[0180] This faceVec ｎ ” By using faceVec, even if the device n cannot perform face sensing, it is possible to reproduce a panel image that correctly reflects the viewpoint of the user U. ｎ ”In the process of calculating the above, all matrices that are multiplied to the left of the faceVec overline are multiplied, and the rotation component of the multiplication result can be associated with faceRot (Roll / Pitch / Yaw) of the integrated viewpoint information D4 shown in FIG.

[0181] The control unit 101 thus obtains facePos and faceRot for the main device and stores them in the integrated viewpoint information D4 shown in FIG.

[0182] <4-8. Shared Information> Next, the shared information D10 will be described. Fig. 51 is a diagram showing the data structure of the shared information D10. For the shared information D10, the control unit 101 (shared information generation unit C6) integrates the connection topology information D3, integrated viewpoint information D4, and device absolute position information D7, and stores the integrated information in the shared information D10 shown in Fig. 51.

[0183] This shared information D10 is the data transmitted from the main device to the sub-device by STATUS_PUSH.

[0184] <<5. Modifications>> Although the embodiment of the present disclosure has been described up to this point, the information processing method according to the present embodiment can be modified in several other ways.

[0185] <5-1. First Modification> First, the first modification will be described. In the first modification, instead of acquiring a face image using an RGB camera in the face sensing unit C1, an optical marker worn around the head of the user U may be used in combination with an infrared camera that detects the marker. In this case, the positions of the head and both eyes are estimated as a three-dimensional rigid body with 6 DoF degrees of freedom.

[0186] <5-2. Second Modification> Next, a second modification will be described. In the second modification, the viewpoint positions of two or more users U may be shared using the communication means of this embodiment. FIG. 52 is an explanatory diagram of a case where two users U face each other to view. FIG. 52 shows a situation where two users U-1 and U-2 hold spatial display devices 100-1 and 100-2, respectively, and face each other to view each other's screen. The spatial display device 100-1 uses the Face of user U-2 as viewpoint information. ２and obtains the viewpoint information Face of user U-2. ２ On the other hand, the spatial display device 100-2 displays a viewpoint video having motion parallax with respect to the user U-2 based on the face of the user U-1 as viewpoint information. １ and obtains the viewpoint information Face of user U-1. １ Based on this, a viewpoint video having motion parallax with respect to the user U-1 is displayed.

[0187] In this situation, it is desirable for each spatial display device 100 to adopt an individually detected viewpoint position rather than a shared viewpoint. In the second modified example, an operation mode for dealing with this case is called an LEP (Local Eye Point) mode.

[0188] Note that Figure 52 shows an example in which spatial display devices 100-1 and 100-2 display characters corresponding to users U-1 and U-2, respectively. In the present disclosure, a "character" is a type of virtual object that can move autonomously, and may be called an avatar or an agent. Figure 52 discloses an example in which spatial display devices 100-1 and 100-2 each display a humanoid character, but the technology of the present disclosure is not limited to this. The characters displayed by spatial display device 100 may be movable objects such as animals or robots.

[0189] 53 is a diagram showing the initial state of the LEP mode. First, user U-1 places the spatial display device 100-1 on a table. Since there are no devices nearby, the spatial display device 100-1 becomes the main device and enters a waiting loop for sub-device participation.

[0190] Since the main device itself cannot take a viewpoint, there is no valid viewpoint that exceeds the threshold in topologyInfo of the connection topology information D3, and an image of the default viewpoint position is displayed on the panel. User U-2 then joins in with spatial display device 100-2, and information sharing according to the second modified example begins.

[0191] The spatial display device 100-1 is the main device, the spatial display device 100-2 is the sub-device, and relative information position detection is performed by UWB. STATUS_REPORT from the sub-device to the main device and STATUS_PUSH from the main device to the sub-device are also performed in accordance with this embodiment, but the following points are different in the case of LEP.

[0192] First, in the LEP mode, a viewpoint video of each spatial display device 100 is generated using the sensor of that device. Therefore, the integrated viewpoint information D4 is not calculated, and instead, the faceDetection included in the STATUS_REPORT of each device is listed and included in the shared information D10a according to the second modified example.

[0193] Fig. 54 is a diagram showing the data structure of shared information D10a according to the second modified example. faceDetectionList is a list in which the deviceId of each device is paired with the face detection information of that device. The entry structure of this list is shown in Fig. 55. Fig. 55 is a diagram showing the entry structure of faceDetectionList.

[0194] Second, in the LEP mode, when generating a viewpoint image, step S211 shown in FIG. 17 or step S708 shown in FIG. 23 is changed to the process of "generating a viewpoint image based on the faceDetection information of the target device shared in the target device / face sensing information D1 or the shared information D10a."

[0195] Third, in the LEP mode, by referencing viewpoint information of other devices shared by STATUS_PUSH, it is possible to know the viewpoint position of user U of the other device, which cannot be detected by the target device. This makes it possible to make a character displayed on spatial display device 100-1 in Fig. 52, for example, perform an action or have a conversation related to the facial position or line of sight of user U-1, who is actually behind the character.

[0196] As an example of a conversation based on the face position of user U-1, spatial display device 100-1 may control a speaker to make a character utter an utterance such as, "Master, please stand right behind me, not behind me on the left." As an example of an operation based on the face position of user U-1, spatial display device 100-1 may change the direction of the character's face so that it faces behind spatial display device 100-1, and operate the character to gaze at the face position of user U-1.

[0197] In the LEP mode, in addition to the user's viewpoint information, the target device may acquire, from another device, position information of real objects that are outside the detection range of the sensor of the target device. By referencing the position information of real objects acquired by the other device, the target device can learn the position information of real objects that the target device cannot detect, such as real objects other than the user, such as a desk or chair behind the target device. This makes it possible, for example, to have a character displayed on the spatial display device 100-1 in FIG. 52 perform an action or have a conversation related to a real object that is outside the detection range of the sensor of the spatial display device 100-1.

[0198] <5-3. Third Modification> Next, a third modification will be described. Figures 56 and 57 are explanatory diagrams (part 1) and (part 2) of an example in which relative positions are sensed using a combination of a light emitter and an RGB camera. In the third modification, each spatial display device 100 includes a light emitter and an RGB camera. The light emitter is provided, for example, on the back of each device.

[0199] In the third modified example, the relative position sensing unit C2 obtains the relative position by capturing an image of the light emitted from the light emitter with an RGB camera. Fig. 56 shows a situation in which the light emitted from the light emitter of the spatial display device 100 "1" is captured by the camera images of the spatial display devices 100 "2" and "4".

[0200] In this case, for example, the light emitter at the top of the housing is time-controlled to emit light, and the RGB cameras of all the devices capture images. Then, the relative positions between the devices are determined from the analysis of the bright spot image and time information. As shown in the left diagram of Figure 57, when the light emitter of a specific device (here, "device 01") emits light, device 04 behind this device 01 captures an image including the light emitter (see the right diagram of Figure 57). By analyzing this image using PnP or the like, it becomes possible to determine the relative position of the 6DoF between the RGB camera and the light emitter. Note that, unlike detection using UWB, the detection results of this third modified example are asymmetric between the devices.

[0201] <5-4. Fourth Modification> Next, a fourth modification will be described. Fig. 58 is an explanatory diagram of linked display using a device equipped with a face sensing unit A1 and a device not equipped with the face sensing unit A1 together. In the fourth modification, the device used for linked display does not need to be equipped with the face sensing unit A1.

[0202] As shown in FIG. 58, a device equipped with a face sensing unit A1 is referred to as "Type-A." A device not equipped with a face sensing unit A1 is referred to as "Type-B." In the example of FIG. 58, for Type-B devices 01 and 02, the Status of faceDetectionStatus is always False in STATUS_REPORT. Alternatively, the corresponding processing may be skipped by allocating a specific value space for deviceId or by providing a separate deviceType to distinguish Type-A / Type-B. Type-B devices can omit the camera block, allowing for reasonable applications.

[0203] <5-5. Other Modifications> Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0204] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0205] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0206] <<6. Hardware Configuration>> The spatial display device 100 according to the embodiment of the present disclosure described above is realized by a computer 1000 having a configuration as shown in Fig. 59, for example. Fig. 59 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the spatial display device 100. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0207] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0208] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0209] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the programs according to this embodiment.

[0210] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0211] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0212] For example, when the computer 1000 functions as a spatial display device, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 101. The programs and data according to the present disclosure are stored in the secondary storage device 1400. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0213] Furthermore, functions realized by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs, conventional circuits, and / or combinations thereof, programmed to realize the described functions, without being limited to the hardware configuration described in FIG. 59 . A processor includes transistors and other circuits and is considered to be a circuit or processing circuitry. A processor may also be a programmed processor that executes a program stored in memory. For example, the processor may realize the functions of the control unit 101 as a programmed processor. Note that the circuit configuration of the main display device according to this embodiment corresponds to an example of a "main circuit configuration," and the circuit configuration of the sub-display device corresponds to an example of a "sub-circuit configuration."

[0214] In this specification, a circuit arrangement, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0215] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0216] <<7. Conclusion>> As described above, according to one embodiment of the present disclosure, the spatial display device 100 (corresponding to an example of an "information processing device") includes a sensing unit 102 (corresponding to an example of a "sensor"), a communication unit A5, and a control unit 101 (corresponding to an example of a "circuit configuration"). The sensing unit 102 is configured to acquire a target device relative position D2 (corresponding to an example of "relative position information") of the main display device with respect to the sub-display device. The communication unit A5 transmits viewpoint information of the user of the main display device to the sub-display device, or transmits target device / face sensing information D1 or Face １ The control unit 101 is configured to receive at least one of the viewpoint information and the relative position information, and the relative position information, and based on the relative position information, acquire device absolute position information D7 indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device, and control the display of the main display device based on shared information D10, D10a or integrated viewpoint information D4 (equivalent to an example of "integrated information") based on the viewpoint information and device absolute position information D7. This enables coordinated control including at least coordinated display using only a plurality of spatial display devices.

[0217] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0218] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0219] Note that the present technology can also be configured as follows: (1) An information processing device comprising: a sensor configured to acquire relative position information of a main display device with respect to a sub-display device; a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device or receive viewpoint information of the user from the sub-display device; and a circuit configuration configured to acquire the viewpoint information and the relative position information, acquire device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device based on the relative position information, and control display of the main display device based on integrated information based on the viewpoint information and the device absolute position information. (2) The information processing device described in (1), wherein the circuit configuration transmits a first request inquiring about the presence or absence of the main display device via the communication unit to a network to which the main display device and the sub-display device are connected, and executes a task as the main display device if no response to the first request has been received after a certain period of time has elapsed. (3) The information processing device according to (2), wherein the circuit configuration, as the main display device, generates connection topology information related to topology management of the main display device and the sub-display device in the network based on the viewpoint information and the device absolute position information. (4) The information processing device according to (3), wherein the circuit configuration, as the main display device, in response to receiving the first request via the communication unit, transmits a response indicating that the main display device has already been set to a sender of the first request via the communication unit. (5) The information processing device according to (3) or (4), wherein, as the main display device, in response to receiving a second request via the communication unit that is a registration request for the sub-display device, registers the sender of the second request as the sub-display device in the connection topology information.(6) The information processing device according to (3), (4), or (5), wherein the circuit configuration, as the main display device, receives reports on the status of the sub-display device from the sub-display device via the communication unit and updates the connection topology information based on the reports. (7) The information processing device according to any one of (3) to (6), wherein the integrated information includes shared information shared between the main display device and the sub-display device, and the circuit configuration, as the main display device, generates the shared information periodically or as needed based on the viewpoint information, the device absolute position information, and the connection topology information, and transmits the shared information via the communication unit to the sub-display device registered in the connection topology information. (8) The information processing device according to (7), wherein the circuit configuration controls display on a 3D panel display unit of the sub-display device by transmitting the shared information to the sub-display device. (9) The information processing device according to any one of (6), (7) or (8), wherein the connection topology information has information on the distance of each of the main display device and the sub display device from the user and on the reliability of acquisition of the viewpoint information, and the circuit configuration selects a new main display device from the main display device and the sub display device registered in the connection topology information based on weighting based on the distance and the reliability. (10) The information processing device according to any one of (1) to (9), wherein the integrated information includes the viewpoint information acquired by each of the main display device and the sub display device, which is shared between the main display device and the sub display device, and the circuit configuration controls the main display device to display a viewpoint video having motion parallax based on the viewpoint information of the user of the sub display device.(11) The information processing device according to (10), wherein the viewpoint image includes a virtual object capable of autonomously operating, and the circuit configuration acquires viewpoint information of a user of the main display device from the sub-display device via the integrated information, and controls the main display device so that the virtual object performs an action corresponding to at least one of a facial position or a line of sight of the user of the main display device based on the viewpoint information of the user of the main display device. (12) The information processing device according to (11), wherein the circuit configuration controls a speaker of the main display device so as to output sound of the virtual object corresponding to a facial position or a line of sight of the user of the main display device based on the viewpoint information of the user of the main display device. (13) The information processing device according to (10), (11), or (12), wherein the viewpoint image includes a virtual object capable of autonomously operating, and the circuit configuration acquires position information of a real object that is outside a detection range of the sensor of the main display device from the sub-display device, and controls the main display device so that the virtual object performs an action corresponding to the position information of the real object. (14) The information processing device according to (13), wherein the circuit configuration controls a speaker of the main display device to output a sound of the virtual object according to position information of the real object. (15) The information processing device according to any one of (1) to (14), wherein the sensor includes an UWB sensor.(16) An information processing device comprising: a sensor configured to acquire relative position information of a sub-display device with respect to a main display device; a communication unit configured to at least one of transmit viewpoint information of a user of the sub-display device to the main display device or receive viewpoint information of the user from the main display device; and a circuit configuration configured to acquire the viewpoint information and the relative position information and transmit them to the main display device via the communication unit, receive from the main display device via the communication unit integrated information based on the viewpoint information and device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device acquired by the main display device based on the relative position information, and control the display of the sub-display device based on the integrated information. (17) The information processing device according to (16), wherein the circuit configuration transmits, via the communication unit, a first request inquiring about the presence or absence of the main display device to a network to which the main display device and the sub-display device are connected, and when a response to the first request is received within a certain period of time, transmits, via the communication unit, to the main display device identified by the response, a second request requesting registration as the sub-display device, thereby receiving registration in connection topology information relating to the topology of the main display device and the sub-display device in the network. (18) The information processing device according to (16) or (17), wherein the circuit configuration receives, via the communication unit, the integration information transmitted periodically or as needed from the main display device, and when the integration information includes identification information identifying the main display device, executes a task as the main display device instead of a task as the sub-display device.(19) An information processing system including a main display device and a sub-display device, wherein the main display device comprises: a sensor configured to acquire relative position information of the main display device with respect to the sub-display device; a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device or receive viewpoint information of the user from the sub-display device; and a main circuit configuration configured to acquire the viewpoint information and the relative position information, acquire device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device based on the relative position information, and control a display of the main display device based on integrated information based on the viewpoint information and the device absolute position information; and the sub-display device comprises: a sensor configured to acquire the relative position information of the sub-display device with respect to the main display device; and a communication unit configured to at least one of transmit viewpoint information of a user of the sub-display device to the main display device or receive viewpoint information of the user from the main display device, and acquire the viewpoint information and the relative position information and transmit them to the main display device via the communication unit, a sub-circuit configuration configured to receive, from the main display device via the communication unit, the integrated information based on the device absolute position information and the viewpoint information acquired by the main display device based on the relative position information, and to control the display of the sub-display device based on the integrated information.(20) An information processing method executed by an information processing device having a sensor configured to acquire relative position information of a main display device with respect to a sub-display device, and a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device or receive viewpoint information of the user from the sub-display device, the information processing method comprising: acquiring the viewpoint information and the relative position information; acquiring device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device, based on the relative position information; and controlling display of the main display device based on integrated information based on the viewpoint information and the device absolute position information.

[0220] 1 Display system 100 Spatial display device 101 Control unit 102 Sensing unit 103 3D panel display unit 200 Wireless router A1 Face sensing unit A2 Relative position sensing unit A3 Information processing unit A4 Video processing unit A5 Communication unit C1 Face sensing unit C10 Slave device information receiving unit C11 Shared information transmitting unit C12 Shared information analysis unit C13 Slave device information generating unit C14 Slave device information transmitting unit C15 Shared information receiving unit C2 Relative position sensing unit C3 Connection topology control unit C4 Integrated viewpoint information generating unit C5 Device position coordinate conversion unit C6 Shared information generating unit C7 Video generating unit C8 Slave device information analyzing unit C9 Video display unit D1 Target device / face sensing information D10 Shared information D10a Shared information D2 Target device relative position D3 Connection topology information D4 Integrated viewpoint information D5 Other device / face sensing information D6 Relative position of other devices D7 Absolute device position information D8 Panel image D9 Slave device information

Claims

1. An information processing device comprising: a sensor configured to acquire relative position information of a main display device with respect to a sub-display device; a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device and receive viewpoint information of the user from the sub-display device; and a circuit configuration configured to acquire the viewpoint information and the relative position information, acquire device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device, based on the relative position information, and control the display of the main display device based on integrated information based on the viewpoint information and the device absolute position information.

2. The information processing device according to claim 1, wherein the circuit configuration transmits a first request, via the communication unit, to a network to which the main display device and the sub-display device are connected, inquiring about the presence or absence of the main display device, and if no response to the first request is received after a certain period of time has elapsed, executes a task as the main display device.

3. The information processing device according to claim 2, wherein the circuit configuration, as the main display device, generates connection topology information relating to topology management of the main display device and the sub-display device in the network based on the viewpoint information and the device absolute position information.

4. The information processing device according to claim 3, wherein the circuit configuration, in response to receiving the first request via the communication unit as the main display device, transmits a response via the communication unit to the sender of the first request indicating that the main display device has already been set.

5. The information processing device according to claim 3, wherein the circuit configuration, in response to receiving a second request, which is a request for registration as the sub-display device via the communication unit, as the main display device, registers the sender of the second request as the sub-display device in the connection topology information.

6. The information processing device according to claim 3, wherein the circuit configuration, as the main display device, receives a report on the status of the sub-display device from the sub-display device via the communication unit, and updates the connection topology information based on the report.

7. An information processing device as described in claim 3, wherein the integrated information includes shared information shared between the main display device and the sub-display device, and the circuit configuration, as the main display device, generates the shared information periodically or at any time based on the viewpoint information, the device absolute position information and the connection topology information, and transmits the shared information via the communication unit to the sub-display device registered in the connection topology information.

8. The information processing device according to claim 7, wherein the circuit configuration controls the display of a 3D panel display unit of the sub-display device by transmitting the shared information to the sub-display device.

9. An information processing device as described in claim 6, wherein the connection topology information includes information regarding the distance of each of the main display device and the sub-display device from the user and the reliability of obtaining the viewpoint information, and the circuit configuration selects a new main display device from among the main display device and the sub-display device registered in the connection topology information based on weighting based on the distance and the reliability.

10. The information processing device described in claim 1, wherein the integrated information includes the viewpoint information acquired by each of the main display device and the sub-display device and shared between the main display device and the sub-display device, and the circuit configuration controls the main display device to display a viewpoint image having motion parallax based on the user of the sub-display device based on the viewpoint information of the user of the sub-display device.

11. The information processing device described in claim 10, wherein the viewpoint image includes a virtual object capable of operating autonomously, and the circuit configuration acquires the viewpoint information of the user of the main display device from the sub-display device via the integrated information, and controls the main display device based on the viewpoint information of the user of the main display device so that the virtual object performs an action corresponding to at least one of the face position or line of sight direction of the user of the main display device.

12. The information processing device according to claim 11, wherein the circuit configuration controls a speaker of the main display device to output sound of the virtual object according to the face position or line of sight direction of the user of the main display device, based on the viewpoint information of the user of the main display device.

13. An information processing device as described in claim 10, wherein the viewpoint image includes a virtual object capable of operating autonomously, and the circuit configuration acquires, from the sub-display device, position information of a real object that is outside the detection range of the sensor of the main display device, and controls the main display device so that the virtual object performs an action according to the position information of the real object.

14. The information processing device according to claim 13, wherein the circuit configuration controls a speaker of the main display device so as to output a sound of the virtual object according to the position information of the real object.

15. The information processing device according to claim 1, wherein the sensor includes an UWB sensor.

16. An information processing device comprising: a sensor configured to acquire relative position information of a sub-display device with respect to a main display device; a communication unit configured to at least one of transmit viewpoint information of a user of the sub-display device to the main display device or receive viewpoint information of the user from the main display device; and a circuit configuration configured to acquire the viewpoint information and the relative position information and transmit it to the main display device via the communication unit, receive from the main display device via the communication unit integrated information based on the viewpoint information and device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device acquired by the main display device based on the relative position information, and control the display of the sub-display device based on the integrated information.

17. The information processing device according to claim 16, wherein the circuit configuration transmits a first request via the communication unit to the network to which the main display device and the sub-display device are connected, inquiring about the presence or absence of the main display device, and if a response to the first request is received within a certain period of time, transmits a second request via the communication unit to the main display device indicated by the response, requesting registration as the sub-display device, thereby receiving registration in connection topology information regarding the topology of the main display device and the sub-display device on the network.

18. The information processing device described in claim 16, wherein the circuit configuration receives the integrated information transmitted periodically or at any time from the main display device via the communication unit, and when the integrated information includes identification information indicating the main display device, executes a task as the main display device instead of a task as the sub-display device.

19. An information processing system including a main display device and a sub-display device, wherein the main display device comprises: a sensor configured to acquire relative position information of the main display device with respect to the sub-display device; a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device or receive viewpoint information of the user from the sub-display device; and a main circuit configuration configured to acquire the viewpoint information and the relative position information, acquire device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device based on the relative position information, and control display of the main display device based on integrated information based on the viewpoint information and the device absolute position information; and the sub-display device comprises: a sensor configured to acquire the relative position information of the sub-display device with respect to the main display device; and a communication unit configured to at least one of transmit viewpoint information of a user of the sub-display device to the main display device or receive viewpoint information of the user from the main display device, and acquire the viewpoint information and the relative position information and transmit them to the main display device via the communication unit, a sub-circuit configuration configured to receive, from the main display device via the communication unit, the integrated information based on the device absolute position information and the viewpoint information acquired by the main display device based on the relative position information, and to control the display of the sub-display device based on the integrated information.

20. An information processing method executed by an information processing device comprising a sensor configured to acquire relative position information of a main display device with respect to a sub-display device, and a communication unit configured to at least one of transmit viewpoint information of a user of the main display device to the sub-display device and receive viewpoint information of the user from the sub-display device, the information processing method comprising: acquiring the viewpoint information and the relative position information; acquiring device absolute position information indicating position information of the main display device and the sub-display device in a coordinate system based on the main display device, based on the relative position information; and controlling display of the main display device based on integrated information based on the viewpoint information and the device absolute position information.

Images